00000100:00100000:2.0:1787695829.738803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145497984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695829.738825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695829.738830:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695829.738833:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695829.738839:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3729 00000100:00080000:2.0:1787695829.738844:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695829.738849:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695829.738856:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695829.738869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145498112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695829.738875:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695829.738878:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695829.738879:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695829.738882:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3729 00000100:00080000:2.0:1787695829.738886:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695829.738891:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695829.738897:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695829.738928:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145498240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695829.738943:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695829.738946:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695829.738948:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695829.738956:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3729 00000100:00080000:2.0:1787695829.738963:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695829.738968:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695829.738974:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695829.738990:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145498368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859041:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859056:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695834.859062:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859075:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859089:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859099:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695834.859148:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145498496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859183:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859187:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695834.859189:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859195:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859199:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859204:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859216:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695834.859239:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145498624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859257:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859265:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695834.859267:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859271:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859275:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859280:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859285:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695834.859306:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145498752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859318:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859322:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695834.859324:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859328:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859333:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859338:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859343:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695834.859404:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145498880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859416:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859421:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695834.859425:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859430:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859435:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859441:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859450:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695834.859468:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145499008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859486:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695834.859488:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859496:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859503:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859513:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859520:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695834.859532:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145499136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859564:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695834.859566:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859572:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859576:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859581:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695834.859601:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145499264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695834.859622:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859627:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859632:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859641:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859648:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695834.859663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145499392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695834.859691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695834.859697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695834.859701:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695834.859710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3734 00000100:00080000:2.0:1787695834.859716:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695834.859724:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695834.859735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695834.859756:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145499520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978456:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978464:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695839.978468:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978476:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978485:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978491:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695839.978523:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145499648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978549:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978552:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695839.978553:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978556:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978559:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978561:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978568:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695839.978576:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145499776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695839.978586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978589:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978591:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978594:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978597:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695839.978603:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145499904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978609:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695839.978615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978622:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978625:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978628:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978631:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695839.978641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145500032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978649:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695839.978650:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978652:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978654:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978657:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978660:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695839.978666:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145500160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978674:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695839.978675:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978677:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978680:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978686:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695839.978699:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145500288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978709:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695839.978713:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978715:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978717:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978719:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695839.978732:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145500416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978740:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695839.978752:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978757:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978760:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978763:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978768:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695839.978779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145500544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695839.978792:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695839.978798:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695839.978802:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695839.978808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3739 00000100:00080000:2.0:1787695839.978812:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695839.978817:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695839.978820:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695839.978830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145500672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787695845.098681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145500800:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695845.098781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.098865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183644000 x1874531145500800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.098905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145500800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.098950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.098964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.098974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145500800:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695845.099017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145500928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695845.099056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183644380 x1874531145500928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145500928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145500928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695845.099116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145501056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695845.099130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183645180 x1874531145501056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145501056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145501056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695845.099157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145501184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695845.099169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183647480 x1874531145501184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145501184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145501184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695845.099194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145501312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695845.099206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183647800 x1874531145501312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145501312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145501312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695845.099233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695845.099245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183646a00 x1874531145501440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145501440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695845.099271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695845.099284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183647b80 x1874531145501568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145501568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695845.099350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695845.099389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183644e00 x1874531145501696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145501696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695845.099420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695845.099431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695845.099438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695845/real 1787695845] req@ffff8f5183644700 x1874531145501824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695845.099445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145501824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695845.099450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695845.099451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695845.099454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145501824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695850.218616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145501952:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695850.218777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695850.218829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695850.218859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695850.218897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695850.218925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695850.218948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695850.218960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695850.218972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695853.290839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f5187389c00 x1874531145502720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.290871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145502720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.290882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.290887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.290892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695853.290901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f5187389880 x1874531145502848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.290916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145502848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.290925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.290929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.290932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695853.290941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f518738b100 x1874531145502976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.290955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145502976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.290964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.290968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.290971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695853.290979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f5187388380 x1874531145502080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.291002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145502080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.291011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.291014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.291018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695853.291026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f518738a300 x1874531145502208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.291039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145502208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.291047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.291050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.291054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695853.291060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f5187389f80 x1874531145502336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.291072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145502336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.291080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.291084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.291087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695853.291095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f518738ad80 x1874531145502464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.291119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145502464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.291127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.291131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.291134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145502464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695853.291141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f5187389500 x1874531145502592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.291160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145502592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.291169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.291172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.291176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145502592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695853.291182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695850/real 1787695853] req@ffff8f518738aa00 x1874531145501952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695853.291194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145501952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695853.291202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695853.291206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695853.291209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145501952:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695855.338741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145503104:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695855.338853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.338936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f5187388700 x1874531145503104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.338974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388700 x1874531145503104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145503104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695855.339160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695855.339240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f5187388000 x1874531145503232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145503232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695855.339315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695855.339347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f518738b480 x1874531145503360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145503360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695855.339507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695855.339534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f518738bb80 x1874531145503488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145503488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695855.339578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695855.339597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f5187388e00 x1874531145503616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145503616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145503616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695855.339648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145503744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695855.339685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f5187389180 x1874531145503744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145503744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145503744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695855.339745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145503872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695855.339768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f518738a680 x1874531145503872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145503872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145503872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695855.339820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145504000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695855.339842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f5187388a80 x1874531145504000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145504000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145504000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695855.339889:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145504128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695855.339913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695855.339925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695855/real 1787695855] req@ffff8f518738b800 x1874531145504128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695855.339936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145504128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695855.339944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695855.339947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695855.339950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145504128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695860.458673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145504256:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695860.458753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.458810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f518738b800 x1874531145504256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.458866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145504256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.458918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.458929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.458941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145504256:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695860.458967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695860.458999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f5187388a80 x1874531145504384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145504384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695860.459109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695860.459123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f518738a680 x1874531145504512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145504512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695860.459153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695860.459164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f5187389180 x1874531145504640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145504640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695860.459191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695860.459200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f5187388e00 x1874531145504768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145504768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145504768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695860.459224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145504896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695860.459234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f518738bb80 x1874531145504896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145504896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145504896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695860.459260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145505024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695860.459271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f518738b480 x1874531145505024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145505024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145505024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695860.459295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145505152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695860.459305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f5187388000 x1874531145505152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145505152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145505152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695860.459330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145505280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695860.459339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695860.459352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695860/real 1787695860] req@ffff8f5187388700 x1874531145505280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695860.459412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388700 x1874531145505280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695860.459418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695860.459420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695860.459422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145505280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695865.578665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145505408:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695865.578788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.578882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183647100 x1874531145505408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.578925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145505408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.578975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145505408:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695865.579065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695865.579139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183646680 x1874531145505536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145505536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695865.579185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695865.579197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183646d80 x1874531145505664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145505664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695865.579225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695865.579236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183645c00 x1874531145505792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145505792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695865.579263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695865.579274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183645500 x1874531145505920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145505920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145505920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695865.579321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695865.579393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183644700 x1874531145506048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145506048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695865.579439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695865.579454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183644e00 x1874531145506176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145506176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695865.579481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695865.579522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183647b80 x1874531145506304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145506304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695865.579577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695865.579588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695865.579595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695865/real 1787695865] req@ffff8f5183646a00 x1874531145506432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695865.579602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145506432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695865.579608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695865.579610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695865.579613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145506432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695870.698675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145506560:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695870.698760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.698826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183647800 x1874531145506560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.698851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145506560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.698862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.698866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.698872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145506560:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695870.698914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145506688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695870.698953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.698981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183647480 x1874531145506688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145506688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145506688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695870.699057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145506816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695870.699087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183645180 x1874531145506816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145506816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145506816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695870.699175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145506944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695870.699203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183644380 x1874531145506944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145506944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145506944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695870.699237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145507072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695870.699255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183644000 x1874531145507072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145507072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145507072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695870.699311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695870.699339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183647800 x1874531145507200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145507200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695870.699429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695870.699446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183646a00 x1874531145507328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145507328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695870.699480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695870.699490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183647b80 x1874531145507456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145507456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695870.699516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695870.699526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695870.699532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695870/real 1787695870] req@ffff8f5183644e00 x1874531145507584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695870.699539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145507584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695870.699544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695870.699546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695870.699548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145507584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695875.818689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145507712:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695875.818775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.818838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f518738aa00 x1874531145507712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.818878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145507712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.818899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.818905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.818912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145507712:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695875.818944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145507840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695875.818970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.818982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f5187389500 x1874531145507840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.818991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145507840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.818997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.818999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145507840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695875.819005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145507968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695875.819016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f518738ad80 x1874531145507968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145507968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145507968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695875.819042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145508096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695875.819054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f5187389f80 x1874531145508096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145508096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145508096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695875.819079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145508224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695875.819089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f518738a300 x1874531145508224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145508224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145508224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695875.819114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695875.819124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f5187388380 x1874531145508352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145508352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695875.819152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695875.819171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f518738b100 x1874531145508480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145508480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695875.819216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695875.819231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f5187389880 x1874531145508608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145508608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695875.819271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695875.819282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695875.819288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695875/real 1787695875] req@ffff8f5187389c00 x1874531145508736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695875.819295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145508736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695875.819300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695875.819302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695875.819304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145508736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695880.938683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145508864:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695880.938772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.938838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f5187389c00 x1874531145508864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.938872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145508864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.938898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.938909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.938919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145508864:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695880.938957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145508992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695880.938993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f5187389880 x1874531145508992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145508992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145508992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695880.939045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145509120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695880.939064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f518738b100 x1874531145509120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145509120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145509120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695880.939116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145509248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695880.939133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f5187388380 x1874531145509248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145509248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145509248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695880.939176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145509376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695880.939197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f518738a300 x1874531145509376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145509376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145509376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695880.939284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695880.939307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f5187389f80 x1874531145509504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145509504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695880.939383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695880.939407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f518738ad80 x1874531145509632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145509632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695880.939455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695880.939488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f5187389500 x1874531145509760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145509760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695880.939553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695880.939574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695880.939586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695880/real 1787695880] req@ffff8f518738aa00 x1874531145509888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695880.939599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145509888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695880.939608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695880.939611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695880.939615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145509888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695886.058714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145510016:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695886.058849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695886.058886:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695886.058915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695886.058943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695886.058962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145510656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695886.058989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145510784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695886.059015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145510912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695886.059034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145511040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695889.130963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183646a00 x1874531145511040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145511040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145511040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695889.131045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183644e00 x1874531145510784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145510784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145510784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695889.131113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183647b80 x1874531145510912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145510912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145510912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695889.131177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183645500 x1874531145510144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145510144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695889.131217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183645c00 x1874531145510272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145510272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695889.131255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183646d80 x1874531145510400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145510400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695889.131293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183646680 x1874531145510528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145510528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145510528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695889.131328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183647100 x1874531145510656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145510656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145510656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695889.131402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695886/real 1787695889] req@ffff8f5183644700 x1874531145510016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695889.131417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145510016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695889.131427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695889.131430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695889.131434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145510016:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695891.178722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145511168:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695891.178805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.178873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f5187388700 x1874531145511168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.178901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388700 x1874531145511168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.178919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.178933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.178948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145511168:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695891.178987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695891.179029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f5187388000 x1874531145511296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145511296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695891.179177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695891.179237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f518738b480 x1874531145511424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145511424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695891.179340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695891.179415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f518738bb80 x1874531145511552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145511552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695891.179497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695891.179543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f5187388e00 x1874531145511680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145511680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145511680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695891.179647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145511808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695891.179673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f5187389180 x1874531145511808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145511808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145511808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695891.179721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145511936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695891.179742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f518738a680 x1874531145511936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145511936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145511936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695891.179801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145512064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695891.179818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f5187388a80 x1874531145512064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145512064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145512064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695891.179847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145512192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695891.179857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695891.179863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695891/real 1787695891] req@ffff8f518738b800 x1874531145512192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695891.179870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145512192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695891.179876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695891.179878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695891.179880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145512192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695896.298629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145512320:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787695896.298696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.298737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f518738b800 x1874531145512320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.298768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145512320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.298789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.298798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.298804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145512320:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787695896.298814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695896.298841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.298856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f5187388a80 x1874531145512448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.298885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145512448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.298892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.298896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.298899:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695896.298906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695896.298918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.298924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f518738a680 x1874531145512576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.298932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145512576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.298937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.298939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.298941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695896.298993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695896.299010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.299018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f5187389180 x1874531145512704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.299026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145512704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.299032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.299034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.299036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695896.299039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695896.299049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.299055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f5187388e00 x1874531145512832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.299062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145512832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.299067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.299069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.299071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145512832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695896.299074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145512960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695896.299084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.299089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f518738bb80 x1874531145512960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.299097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145512960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.299102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.299103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.299105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145512960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695896.299108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145513088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787695896.299117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.299123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f518738b480 x1874531145513088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.299130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145513088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.299140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.299142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.299144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145513088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787695896.299147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145513216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695896.299160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.299170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f5187388000 x1874531145513216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.299182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145513216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.299190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.299194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.299197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145513216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787695896.299203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145513344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787695896.299223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787695896.299232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695896/real 1787695896] req@ffff8f5187388700 x1874531145513344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787695896.299240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388700 x1874531145513344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787695896.299245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787695896.299247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787695896.299250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145513344:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787695901.418521:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418530:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695901.418535:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418545:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418557:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418564:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695901.418629:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145513472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418656:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418659:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695901.418660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418669:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418672:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418676:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418686:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695901.418697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145513600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418707:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418714:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695901.418717:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418721:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418723:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418727:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695901.418739:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145513728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418745:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418748:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695901.418749:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418752:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418754:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418756:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418759:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695901.418766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145513856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418773:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695901.418774:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418777:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418780:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418783:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418785:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695901.418796:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145513984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418806:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418813:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695901.418814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418817:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418819:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418822:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695901.418835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145514112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418844:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418848:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695901.418850:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418854:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418858:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418863:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418881:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695901.418895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145514240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418915:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418922:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695901.418924:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418929:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418934:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.418940:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.418948:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695901.418962:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145514368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695901.418978:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695901.418983:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695901.418987:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695901.418995:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3801 00000100:00080000:2.0:1787695901.418997:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695901.419000:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695901.419003:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695901.419011:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145514496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538438:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538460:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695906.538465:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538479:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538492:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538501:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695906.538549:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145514624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538575:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538578:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695906.538579:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538583:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538588:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538593:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.538619:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695906.538638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145514752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538649:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538653:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695906.538656:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538661:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538666:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538670:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.538674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695906.538686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145514880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538698:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695906.538700:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538710:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538713:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.538717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695906.538726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145515008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538738:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538740:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695906.538742:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538748:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538751:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.538754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695906.538770:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145515136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538793:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538797:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695906.538802:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538813:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538820:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538829:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.538838:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695906.538872:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145515264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.538903:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.538910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695906.538914:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.538924:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.538931:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.538941:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.538950:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695906.538970:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145515392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.539003:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.539007:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695906.539011:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.539019:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.539026:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.539036:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.539050:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695906.539078:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145515520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695906.539098:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695906.539102:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695906.539105:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695906.539112:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3806 00000100:00080000:2.0:1787695906.539121:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695906.539135:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695906.539144:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695906.539163:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145515648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378493:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378505:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695942.378509:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378520:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378532:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378540:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695942.378580:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145522688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378605:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378608:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695942.378609:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378613:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378616:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378620:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695942.378647:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145522816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378662:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695942.378669:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378674:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378678:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378683:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378688:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695942.378699:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145522944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378709:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695942.378710:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378713:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378716:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378718:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378722:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695942.378733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145523072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378744:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695942.378745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378747:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378749:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378752:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695942.378761:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145523200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378767:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378769:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695942.378773:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378779:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378784:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378790:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695942.378805:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145523328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378830:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695942.378834:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378842:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378847:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378850:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378853:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695942.378860:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145523456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378873:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695942.378874:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378878:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378880:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378883:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378887:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695942.378898:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145523584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695942.378907:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695942.378914:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695942.378918:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695942.378926:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3842 00000100:00080000:2.0:1787695942.378931:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695942.378938:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695942.378946:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695942.378956:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145523712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738515:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738528:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695957.738533:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738545:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738559:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738567:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695957.738610:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145526144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738646:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695957.738648:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738652:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738656:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738660:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.738671:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695957.738685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145526272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695957.738705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738712:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738720:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.738732:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695957.738746:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145526400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695957.738764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738769:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738774:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738778:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.738784:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695957.738798:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145526528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738814:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695957.738821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738827:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738831:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738835:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.738841:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695957.738854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145526656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738864:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738869:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695957.738874:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738884:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738889:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738894:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.738938:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695957.738958:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145526784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.738969:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.738973:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695957.738975:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.738979:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.738984:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.738992:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.739000:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695957.739016:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145526912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.739030:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.739036:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695957.739038:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.739045:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.739052:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.739062:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.739069:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695957.739085:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145527040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695957.739107:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695957.739115:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695957.739117:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695957.739125:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3857 00000100:00080000:2.0:1787695957.739131:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695957.739136:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695957.739142:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695957.739154:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145527168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218468:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695978.218480:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218488:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218497:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218510:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695978.218570:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145530752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695978.218596:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218638:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218647:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218663:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695978.218679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145530880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695978.218692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218695:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218699:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218702:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218705:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695978.218715:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145531008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695978.218732:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218735:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218737:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218740:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218743:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695978.218752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145531136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218762:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218769:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695978.218773:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218778:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218781:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218786:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218789:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695978.218801:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145531264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218816:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695978.218823:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218831:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218834:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218836:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695978.218846:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145531392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218857:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218860:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695978.218861:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218864:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218867:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218873:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218877:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695978.218889:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145531520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218909:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218914:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695978.218916:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218925:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.218931:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.218941:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.218949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695978.218964:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145531648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695978.218981:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695978.218986:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695978.218989:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695978.218997:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3877 00000100:00080000:2.0:1787695978.219004:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695978.219013:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695978.219022:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695978.219041:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145531776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458469:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458479:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787695988.458484:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458495:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458505:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458513:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787695988.458549:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145533056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458578:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695988.458584:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458589:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458608:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695988.458622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145533184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695988.458647:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458653:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458657:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458661:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458667:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695988.458685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145533312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458704:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695988.458706:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458711:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458716:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458720:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458726:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695988.458742:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145533440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458752:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458756:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695988.458758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458768:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458773:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458778:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695988.458792:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145533568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458805:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458810:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787695988.458813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458818:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458822:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458826:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458832:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695988.458842:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145533696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458852:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458855:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787695988.458857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458860:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458864:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458867:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458872:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787695988.458883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145533824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458891:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458894:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787695988.458896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458900:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458903:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458907:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458911:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695988.458921:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145533952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787695988.458931:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787695988.458935:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787695988.458936:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787695988.458939:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 3888 00000100:00080000:2.0:1787695988.458942:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787695988.458946:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787695988.458950:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787695988.458959:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145534080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696055.018604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145548032:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696055.018659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.018705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183647800 x1874531145548032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.018724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145548032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.018733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.018739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.018745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145548032:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696055.018767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696055.018791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.018802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183644000 x1874531145548160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.018819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145548160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.018826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.018831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.018837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696055.018846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696055.018862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.018872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183644380 x1874531145548288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.018886:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145548288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.018901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.018909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.018915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696055.018925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696055.018949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.018964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183645180 x1874531145548416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.018972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145548416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.018996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.018999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.019005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696055.019022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696055.019041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.019051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183647480 x1874531145548544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.019067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145548544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.019072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.019081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.019086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145548544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696055.019092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145548672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696055.019110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.019121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183647800 x1874531145548672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.019132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145548672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.019140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.019143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.019146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145548672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696055.019151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145548800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696055.019169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.019176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183644700 x1874531145548800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.019183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145548800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.019188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.019190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.019192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145548800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696055.019195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145548928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696055.019204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.019210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183647100 x1874531145548928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.019216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145548928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.019244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.019246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.019248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145548928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696055.019259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145549056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696055.019271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696055.019277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696055/real 1787696055] req@ffff8f5183646680 x1874531145549056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696055.019284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145549056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696055.019288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696055.019290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696055.019292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145549056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696060.138568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145549184:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696060.138623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.138663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f518738aa00 x1874531145549184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.138685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145549184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.138699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.138706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.138713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145549184:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696060.138723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696060.138746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.138758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f5187389500 x1874531145549312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.138778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145549312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.138789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.138794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.138798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696060.138824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696060.138835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.138841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f518738ad80 x1874531145549440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.138848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145549440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.138852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.138854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.138856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696060.138859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696060.138869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.138874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f5187389f80 x1874531145549568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.138880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145549568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.138886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.138888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.138891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696060.138896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696060.138912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.138923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f518738a300 x1874531145549696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.138933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145549696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.138939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.138942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.138945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145549696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696060.138950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145549824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696060.138966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.138977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f5187388380 x1874531145549824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.138984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145549824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.138989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.138991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.138993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145549824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696060.138996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145549952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696060.139007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.139012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f518738b100 x1874531145549952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.139019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145549952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.139023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.139025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.139026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145549952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696060.139029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145550080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696060.139037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.139042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f5187389880 x1874531145550080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.139048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145550080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.139056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.139058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.139059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145550080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696060.139062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145550208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696060.139070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696060.139074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696060/real 1787696060] req@ffff8f5187389c00 x1874531145550208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696060.139080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145550208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696060.139084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696060.139085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696060.139087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145550208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696065.259572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145550336:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696065.259640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.259691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183646d80 x1874531145550336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.259736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145550336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.259757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.259765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.259772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145550336:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696065.259797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696065.259839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.259855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183645c00 x1874531145550464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.259873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145550464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.259887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.259892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.259896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696065.259904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696065.259920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.259934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183645500 x1874531145550592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.259953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145550592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.259978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.259984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.259988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696065.259994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696065.260017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.260039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183647b80 x1874531145550720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.260096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145550720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.260107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.260111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.260116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696065.260123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696065.260145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.260158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183644e00 x1874531145550848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.260170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145550848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.260175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.260178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.260180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145550848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696065.260194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145550976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696065.260213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.260226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183646a00 x1874531145550976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.260239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145550976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.260256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.260259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.260263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145550976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696065.260269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145551104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696065.260290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.260303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183646d80 x1874531145551104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.260316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145551104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.260324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.260327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.260331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145551104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696065.260336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145551232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696065.260394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.260407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183646680 x1874531145551232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.260416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145551232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.260421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.260423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.260426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145551232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696065.260431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145551360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696065.260445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696065.260451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696065/real 1787696065] req@ffff8f5183647100 x1874531145551360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696065.260458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145551360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696065.260463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696065.260465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696065.260467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145551360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696070.378634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145551488:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696070.378692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.378747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f5187389c00 x1874531145551488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.378786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145551488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.378800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.378807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.378814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145551488:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696070.378854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145551616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696070.378900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.378922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f5187389880 x1874531145551616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.378932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145551616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.378938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.378940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.378943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145551616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696070.378947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145551744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696070.378987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.378999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f518738b100 x1874531145551744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145551744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145551744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696070.379022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145551872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696070.379036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.379043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f5187388380 x1874531145551872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145551872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145551872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696070.379063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145552000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696070.379072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.379078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f518738a300 x1874531145552000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145552000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145552000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696070.379096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696070.379105:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.379111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f5187389f80 x1874531145552128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145552128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696070.379130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696070.379139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.379145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f518738ad80 x1874531145552256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145552256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696070.379173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696070.379183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.379189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f5187389500 x1874531145552384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145552384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696070.379208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696070.379217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696070.379223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696070/real 1787696070] req@ffff8f518738aa00 x1874531145552512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696070.379230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145552512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696070.379235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696070.379237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696070.379238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145552512:192.168.201.125@tcp:8:kworker.0 00010000:00010000:2.0:1787696075.370537:0:55697:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1787696075.498582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145552768:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696075.498658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.498711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f5187388700 x1874531145552768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.498743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388700 x1874531145552768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.498769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.498778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.498789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145552768:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696075.498809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145552896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696075.498848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.498900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f518738a680 x1874531145552896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.498916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145552896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.498923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.498925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.498927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145552896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696075.498946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145553024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696075.498962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.498970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f5187388a80 x1874531145553024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.498977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145553024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.498983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.498985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.498987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145553024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696075.498990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145553152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696075.499000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.499006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f5187388000 x1874531145553152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.499013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145553152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.499018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.499019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.499022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145553152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696075.499026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145553280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696075.499035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.499041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f518738b480 x1874531145553280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.499048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145553280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.499053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.499054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.499056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145553280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696075.499059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696075.499069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.499074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f518738bb80 x1874531145553408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.499081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145553408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.499086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.499089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.499090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696075.499094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696075.499103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.499109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f518738b800 x1874531145553536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.499116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145553536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.499121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.499123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.499125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696075.499128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696075.499138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.499143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f5187389180 x1874531145553664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.499150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145553664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.499155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.499157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.499158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696075.499161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696075.499170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696075.499176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696075/real 1787696075] req@ffff8f5187388e00 x1874531145553792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696075.499182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145553792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696075.499187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696075.499189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696075.499191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145553792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696080.618588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145553920:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696080.618654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.618707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f5187388e00 x1874531145553920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.618751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145553920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.618772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.618792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.618799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145553920:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696080.618829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696080.618864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.618885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f5187389180 x1874531145554048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.618912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145554048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.618937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.618947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.618955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696080.618971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696080.618993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f518738b800 x1874531145554176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145554176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696080.619038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696080.619055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f518738bb80 x1874531145554304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145554304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696080.619098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696080.619113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f518738b480 x1874531145554432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145554432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145554432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696080.619165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696080.619182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f5187388000 x1874531145554560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145554560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696080.619236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696080.619253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f5187388a80 x1874531145554688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145554688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696080.619303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696080.619319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f518738a680 x1874531145554816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145554816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696080.619386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696080.619405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696080.619415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696080/real 1787696080] req@ffff8f5187388700 x1874531145554944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696080.619438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388700 x1874531145554944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696080.619447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696080.619450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696080.619454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145554944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696085.738601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145555072:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696085.738710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696085.738763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696085.738798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696085.738824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696085.738852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145555712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696085.738876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145555840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696085.738906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145555968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696085.738942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145556096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696088.810896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183646d80 x1874531145556096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.810943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145556096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.810959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.810967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.810972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145556096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696088.810982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183646680 x1874531145555968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.810998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145555968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145555968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696088.811037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183644000 x1874531145555712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145555712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145555712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696088.811082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183647100 x1874531145555840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145555840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145555840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696088.811118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183647480 x1874531145555328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145555328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696088.811155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183645180 x1874531145555456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145555456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696088.811203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183644380 x1874531145555584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145555584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696088.811238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183644700 x1874531145555072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145555072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145555072:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696088.811276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696085/real 1787696088] req@ffff8f5183647800 x1874531145555200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696088.811287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145555200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696088.811296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696088.811299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696088.811303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145555200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696090.858630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145556224:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696090.858708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.858758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f518738aa00 x1874531145556224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.858790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145556224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.858805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.858816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.858826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145556224:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696090.858860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696090.858898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.858918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f5187389500 x1874531145556352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.858946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145556352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.858970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.858979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.858988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696090.859003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696090.859037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f518738ad80 x1874531145556480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145556480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696090.859124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696090.859150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f5187389f80 x1874531145556608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145556608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696090.859200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696090.859220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f518738a300 x1874531145556736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145556736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145556736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696090.859291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145556864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696090.859313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f5187388380 x1874531145556864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145556864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145556864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696090.859394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145556992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696090.859425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f518738b100 x1874531145556992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145556992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145556992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696090.859473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145557120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696090.859495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f5187389880 x1874531145557120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145557120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145557120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696090.859541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145557248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696090.859568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696090.859580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff8f5187389c00 x1874531145557248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696090.859593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145557248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696090.859612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696090.859616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696090.859619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145557248:192.168.201.125@tcp:8:kworker.0 00010000:00010000:2.0:1787696095.850615:0:55697:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 16/16 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1787696095.978754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145557504:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696095.978847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.978919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace93800 x1874531145557504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.978972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145557504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145557504:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696095.979051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145557632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696095.979082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace93100 x1874531145557632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145557632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145557632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696095.979140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145557760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696095.979151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace93b80 x1874531145557760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145557760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145557760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696095.979178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145557888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696095.979190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace90e00 x1874531145557888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145557888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145557888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696095.979221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145558016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696095.979231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace91180 x1874531145558016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145558016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145558016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696095.979257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696095.979267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace90700 x1874531145558144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145558144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696095.979297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696095.979307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace92300 x1874531145558272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145558272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696095.979345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696095.979391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace90380 x1874531145558400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145558400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696095.979422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696095.979434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696095.979441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff8f51ace92680 x1874531145558528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696095.979448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145558528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696095.979453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696095.979455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696095.979456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145558528:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696101.098509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098517:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696101.098520:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098529:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098537:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098543:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696101.098576:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145558656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098601:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098603:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696101.098605:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098608:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098611:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098615:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696101.098630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145558784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098639:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098642:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696101.098645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098652:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098657:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098660:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696101.098669:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145558912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098675:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098678:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696101.098679:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098682:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098684:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098687:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696101.098705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145559040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098717:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098720:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696101.098721:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098725:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098728:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098730:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098733:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696101.098740:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145559168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696101.098750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098752:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098754:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098757:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098760:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696101.098769:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145559296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098800:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098803:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696101.098805:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098808:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098810:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098815:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098820:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696101.098831:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145559424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098855:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098858:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696101.098862:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098867:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098870:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098880:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098887:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696101.098897:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145559552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696101.098906:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696101.098909:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696101.098913:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696101.098920:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4000 00000100:00080000:2.0:1787696101.098922:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696101.098925:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696101.098928:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696101.098950:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145559680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218469:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218481:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696106.218486:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218506:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.218518:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.218526:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696106.218568:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145559808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218597:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218601:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696106.218603:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218608:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.218612:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.218617:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.218629:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696106.218658:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145559936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218675:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218688:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696106.218690:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218694:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.218698:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.218702:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.218708:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696106.218720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145560064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696106.218738:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218743:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.218754:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.218776:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.218784:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696106.218802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145560192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218840:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696106.218856:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218861:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.218866:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.218874:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.218886:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696106.218910:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145560320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218934:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218940:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696106.218944:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218952:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.218956:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.218960:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.218965:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696106.218976:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145560448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.218983:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.218985:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696106.218987:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.218996:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.219000:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.219010:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.219018:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696106.219033:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145560576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.219044:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.219057:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696106.219062:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.219074:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.219086:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.219096:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.219113:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696106.219138:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145560704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696106.219167:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696106.219170:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696106.219172:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696106.219187:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4005 00000100:00080000:2.0:1787696106.219196:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696106.219206:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696106.219223:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696106.219246:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145560832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787696135.786498:0:55697:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 4/4 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1787696135.786536:0:55697:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 8/12 locks to regular blp list, 2 blwis in pool 00000100:00080000:2.0:1787696135.787141:0:2402:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8f5187388700 x1874531145566848/t0(0) o103->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:17/18 lens 344/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.938599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145566976:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696136.938661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.938724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183646a00 x1874531145566976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.938773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145566976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.938795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.938818:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.938827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145566976:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696136.938858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696136.938888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.938903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183644e00 x1874531145567104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.938918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145567104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.938925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.938930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.938936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696136.938947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696136.938973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.938993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183647b80 x1874531145567232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145567232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696136.939043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696136.939059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.939067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183645500 x1874531145567360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145567360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696136.939087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696136.939097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.939103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183645c00 x1874531145567488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145567488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145567488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696136.939122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145567616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696136.939132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.939137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183647800 x1874531145567616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145567616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145567616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696136.939175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145567744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696136.939186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.939192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183644700 x1874531145567744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145567744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145567744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696136.939211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145567872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696136.939220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.939226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183644380 x1874531145567872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145567872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145567872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696136.939246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145568000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696136.939255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696136.939260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff8f5183645180 x1874531145568000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696136.939267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145568000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696136.939272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696136.939274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696136.939276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145568000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696142.058582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145568128:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696142.058655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.058734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f518738a680 x1874531145568128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.058761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145568128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.058782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.058789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.058799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145568128:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696142.058813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696142.058850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.058872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f5187388a80 x1874531145568256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.058887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145568256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.058895:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.058899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.058902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696142.058927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696142.058962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.058974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f5187388000 x1874531145568384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.058987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145568384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.058995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.058998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696142.059007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696142.059025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.059035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f518738b480 x1874531145568512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.059046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145568512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.059054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.059058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696142.059067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696142.059082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.059092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f518738bb80 x1874531145568640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.059104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145568640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.059112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.059115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145568640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696142.059124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145568768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696142.059142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.059152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f518738b800 x1874531145568768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.059164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145568768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.059173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.059176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145568768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696142.059186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145568896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696142.059202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.059212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f5187389180 x1874531145568896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.059222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145568896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.059230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.059233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145568896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696142.059241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145569024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696142.059258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.059268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f5187388e00 x1874531145569024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.059280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145569024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.059287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.059290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145569024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696142.059300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145569152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696142.059315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696142.059325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696142/real 1787696142] req@ffff8f518738aa00 x1874531145569152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696142.059337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145569152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696142.059345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696142.059348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696142.059351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145569152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696147.178585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145569280:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696147.178670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.178727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f5187389500 x1874531145569280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.178759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145569280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.178777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.178786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.178795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145569280:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696147.178810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696147.178840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.178860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f518738ad80 x1874531145569408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.178887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145569408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.178904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.178908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.178912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696147.178934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696147.178954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.178964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f5187389f80 x1874531145569536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.178976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145569536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.178984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.178988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.178991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696147.178997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696147.179014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.179023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f518738a300 x1874531145569664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.179036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145569664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.179044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.179048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.179052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696147.179058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696147.179073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.179083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f5187388380 x1874531145569792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.179095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145569792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.179103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.179106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.179109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145569792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696147.179116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145569920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696147.179134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.179145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f518738b100 x1874531145569920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.179156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145569920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.179166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.179169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.179172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145569920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696147.179178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145570048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696147.179194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.179204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f5187389880 x1874531145570048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.179216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145570048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.179224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.179227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.179230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145570048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696147.179236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145570176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696147.179252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.179274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f5187389c00 x1874531145570176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.179286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145570176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.179294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.179297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.179300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145570176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696147.179306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145570304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696147.179324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696147.179333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696147/real 1787696147] req@ffff8f518738aa00 x1874531145570304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696147.179345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145570304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696147.179378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696147.179382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696147.179386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145570304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696152.298640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145570432:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696152.298736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696152.298763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696152.298785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696152.298813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696152.298868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696152.298909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696152.298958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696152.299025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696155.372114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183647800 x1874531145571456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145571456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696155.372177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183644700 x1874531145571328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145571328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696155.372210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183644380 x1874531145571200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145571200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696155.372245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183645180 x1874531145571072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145571072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145571072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696155.372281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183646d80 x1874531145570944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145570944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696155.372313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183647100 x1874531145570560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145570560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696155.372343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183644000 x1874531145570688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145570688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696155.372402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183646680 x1874531145570816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145570816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145570816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696155.372430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696152/real 1787696155] req@ffff8f5183647480 x1874531145570432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696155.372441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145570432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696155.372448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696155.372451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696155.372454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145570432:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696157.418587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145571584:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696157.418667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.418719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183645c00 x1874531145571584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.418761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145571584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.418795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.418806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.418816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145571584:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696157.418846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145571712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696157.418883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.418902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183645500 x1874531145571712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.418933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145571712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.418948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.418956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.418962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145571712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696157.418969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145571840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696157.419005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183647b80 x1874531145571840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145571840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145571840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696157.419056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145571968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696157.419075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183644e00 x1874531145571968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145571968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145571968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696157.419117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145572096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696157.419133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183646a00 x1874531145572096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145572096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145572096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696157.419174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696157.419190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183647480 x1874531145572224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145572224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696157.419241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696157.419257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183646680 x1874531145572352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145572352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696157.419296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696157.419313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183644000 x1874531145572480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145572480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696157.419384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696157.419404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696157.419415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696157/real 1787696157] req@ffff8f5183647100 x1874531145572608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696157.419426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145572608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696157.419433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696157.419436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696157.419439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145572608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696162.538608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145572736:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696162.538663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.538711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f5187388e00 x1874531145572736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.538729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145572736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.538746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.538749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.538753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145572736:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696162.538772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145572864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696162.538792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.538804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f5187389180 x1874531145572864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.538814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145572864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.538828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.538831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.538834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145572864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696162.538840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145572992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696162.538856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.538866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f518738b800 x1874531145572992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.538876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145572992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.538883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.538885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.538888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145572992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696162.538894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145573120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696162.538920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.538931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f518738bb80 x1874531145573120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.538938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145573120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.538943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.538945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.538946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145573120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696162.538950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145573248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696162.538965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.538973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f518738b480 x1874531145573248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.538980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145573248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.538984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.538986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.538988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145573248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696162.538991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696162.539000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.539006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f5187388000 x1874531145573376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.539012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145573376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.539020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.539022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.539024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696162.539035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696162.539043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.539049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f5187388a80 x1874531145573504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.539055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145573504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.539059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.539060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.539062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696162.539065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696162.539073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.539078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f518738a680 x1874531145573632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.539084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145573632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.539088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.539089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.539092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696162.539100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696162.539109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696162.539115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696162/real 1787696162] req@ffff8f5187388e00 x1874531145573760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696162.539121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145573760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696162.539125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696162.539127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696162.539128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145573760:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696167.658490:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696167.658503:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658512:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658523:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658530:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696167.658565:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145573888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658594:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658598:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696167.658599:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658604:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658608:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658613:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.658624:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696167.658638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145574016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658651:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658654:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696167.658657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658660:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658665:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658670:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.658675:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696167.658689:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145574144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696167.658705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658709:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658714:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658721:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.658730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696167.658744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145574272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658777:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696167.658781:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658787:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658791:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658796:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.658801:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696167.658817:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145574400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658841:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696167.658844:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658851:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658859:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658865:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.658874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696167.658892:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145574528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658908:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658915:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696167.658920:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658928:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658935:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.658943:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.658954:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696167.658968:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145574656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.658979:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.658983:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696167.658985:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.658993:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.658999:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.659003:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.659009:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696167.659022:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145574784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696167.659035:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696167.659045:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696167.659048:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696167.659057:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4067 00000100:00080000:2.0:1787696167.659063:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696167.659079:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696167.659094:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696167.659111:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145574912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696172.778794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145575040:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696172.778873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.778955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183646d80 x1874531145575040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.778998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145575040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145575040:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696172.779089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696172.779123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183645180 x1874531145575168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145575168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696172.779168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696172.779181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183644380 x1874531145575296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145575296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696172.779259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696172.779279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183644700 x1874531145575424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145575424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696172.779318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696172.779335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183647800 x1874531145575552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145575552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145575552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696172.779425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145575680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696172.779446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183647100 x1874531145575680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145575680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145575680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696172.779481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145575808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696172.779495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183644000 x1874531145575808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145575808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145575808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696172.779518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145575936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696172.779527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183646680 x1874531145575936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145575936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145575936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696172.779549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145576064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696172.779558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696172.779563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696172/real 1787696172] req@ffff8f5183647480 x1874531145576064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696172.779569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145576064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696172.779573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696172.779575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696172.779577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145576064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696177.898740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145576192:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696177.898841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.898901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f518738aa00 x1874531145576192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.898930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145576192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.898942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.898946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.898950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145576192:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696177.898958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696177.898982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f5187389c00 x1874531145576320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145576320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696177.899035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696177.899056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f5187389880 x1874531145576448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145576448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696177.899151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696177.899173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f518738b100 x1874531145576576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145576576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696177.899217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696177.899236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f5187388380 x1874531145576704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145576704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145576704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696177.899277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145576832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696177.899296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f518738a300 x1874531145576832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145576832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145576832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696177.899338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145576960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696177.899393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f5187389f80 x1874531145576960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145576960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145576960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696177.899438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145577088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696177.899456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f518738ad80 x1874531145577088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145577088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145577088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696177.899500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145577216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696177.899515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696177.899525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696177/real 1787696177] req@ffff8f5187389500 x1874531145577216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696177.899536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145577216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696177.899544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696177.899547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696177.899550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145577216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696183.018578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145577344:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696183.018644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.018687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace92d80 x1874531145577344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.018707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145577344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.018730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.018738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.018743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145577344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696183.018769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696183.018793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.018807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace91880 x1874531145577472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.018817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145577472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.018826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.018832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.018837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696183.018844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696183.018867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.018878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace91c00 x1874531145577600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.018892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145577600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.018907:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.018915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.018921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696183.018933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696183.018961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.018974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace93480 x1874531145577728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.018988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145577728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.018995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.018998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.019001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696183.019006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696183.019036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.019046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace90000 x1874531145577856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.019053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145577856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.019058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.019060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.019062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145577856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696183.019065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145577984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696183.019076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.019081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace91500 x1874531145577984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.019087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145577984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.019096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.019098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.019099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145577984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696183.019110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145578112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696183.019119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.019124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace92d80 x1874531145578112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.019130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145578112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.019135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.019136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.019138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145578112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696183.019141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145578240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696183.019149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.019154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace92680 x1874531145578240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.019160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145578240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.019164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.019165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.019168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145578240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696183.019171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145578368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696183.019178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696183.019183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696183/real 1787696183] req@ffff8f51ace90380 x1874531145578368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696183.019195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145578368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696183.019199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696183.019201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696183.019203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145578368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696188.138713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145578496:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696188.138805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.138881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183646a00 x1874531145578496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.138919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145578496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.138947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.138951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.138956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145578496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696188.138966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145578624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696188.138997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183644e00 x1874531145578624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145578624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145578624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696188.139058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145578752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696188.139072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183647b80 x1874531145578752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145578752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145578752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696188.139100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145578880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696188.139111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183645500 x1874531145578880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145578880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145578880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696188.139138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145579008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696188.139147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183645c00 x1874531145579008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145579008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145579008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696188.139172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696188.139183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183647480 x1874531145579136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145579136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696188.139209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696188.139223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183646680 x1874531145579264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145579264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696188.139269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696188.139291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183644000 x1874531145579392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145579392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696188.139338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696188.139386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696188.139399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696188/real 1787696188] req@ffff8f5183647100 x1874531145579520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696188.139409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145579520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696188.139417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696188.139420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696188.139423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145579520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696193.258628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145579648:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696193.258690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.258738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace92300 x1874531145579648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.258767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145579648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.258789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.258799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.258809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145579648:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696193.258827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145579776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696193.258875:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.258898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace90700 x1874531145579776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.258923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145579776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.258938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.258946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.258953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145579776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696193.259000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145579904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696193.259030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace91180 x1874531145579904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145579904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145579904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696193.259068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145580032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696193.259080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace90e00 x1874531145580032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145580032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145580032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696193.259107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145580160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696193.259117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace93b80 x1874531145580160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145580160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145580160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696193.259142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696193.259151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace93100 x1874531145580288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145580288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696193.259176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696193.259186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace93800 x1874531145580416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145580416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696193.259214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696193.259232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace90380 x1874531145580544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145580544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696193.259285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696193.259300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696193.259307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696193/real 1787696193] req@ffff8f51ace92680 x1874531145580672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696193.259314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145580672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696193.259320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696193.259322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696193.259324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145580672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696198.378631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145580800:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696198.378720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.378803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f5187389500 x1874531145580800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.378852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145580800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.378868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.378877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.378887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145580800:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696198.378925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145580928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696198.378961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.378984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f518738ad80 x1874531145580928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145580928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145580928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696198.379139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145581056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696198.379166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f5187389f80 x1874531145581056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145581056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145581056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696198.379295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145581184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696198.379337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f518738a300 x1874531145581184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145581184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145581184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696198.379396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145581312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696198.379410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f5187388380 x1874531145581312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145581312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145581312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696198.379461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696198.379484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f518738b100 x1874531145581440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145581440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696198.379530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696198.379546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f5187389880 x1874531145581568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145581568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696198.379580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696198.379591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f5187389c00 x1874531145581696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145581696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696198.379616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696198.379626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696198.379632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696198/real 1787696198] req@ffff8f518738aa00 x1874531145581824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696198.379639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145581824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696198.379644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696198.379646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696198.379649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145581824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696203.498623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145581952:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696203.498710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.498764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f5187388e00 x1874531145581952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.498797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145581952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.498829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.498838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.498849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145581952:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696203.498866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696203.498899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.498928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f518738a680 x1874531145582080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.498943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145582080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.498953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.498956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.498960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696203.498986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696203.499009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f5187388a80 x1874531145582208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145582208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696203.499056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696203.499076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f5187388000 x1874531145582336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145582336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696203.499132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696203.499150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f518738b480 x1874531145582464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145582464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145582464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696203.499205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696203.499226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f518738bb80 x1874531145582592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145582592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696203.499273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696203.499291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f518738b800 x1874531145582720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145582720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696203.499347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696203.499393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f5187389180 x1874531145582848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145582848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696203.499440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696203.499460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696203.499472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696203/real 1787696203] req@ffff8f518738aa00 x1874531145582976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696203.499484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145582976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696203.499493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696203.499496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696203.499500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145582976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696208.618749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145583104:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696208.618835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.618896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183647800 x1874531145583104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.618917:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145583104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.618929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.618934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.618940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145583104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696208.618970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696208.618994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183644700 x1874531145583232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145583232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696208.619043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696208.619066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183644380 x1874531145583360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145583360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696208.619107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696208.619122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183645180 x1874531145583488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145583488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696208.619149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696208.619159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183646d80 x1874531145583616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145583616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145583616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696208.619186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145583744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696208.619206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183647100 x1874531145583744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145583744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145583744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696208.619234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145583872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696208.619245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183644000 x1874531145583872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145583872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145583872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696208.619280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145584000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696208.619291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183646680 x1874531145584000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145584000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145584000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696208.619318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145584128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696208.619335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696208.619347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696208/real 1787696208] req@ffff8f5183647480 x1874531145584128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696208.619390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145584128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696208.619396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696208.619398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696208.619401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145584128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696213.738722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145584256:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696213.738814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.738870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183645c00 x1874531145584256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.738915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145584256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.738963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.738974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.738984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145584256:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696213.739030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696213.739071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183645500 x1874531145584384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145584384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696213.739188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696213.739225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183647b80 x1874531145584512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145584512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696213.739351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696213.739437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183644e00 x1874531145584640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145584640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696213.739514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696213.739536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183646a00 x1874531145584768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145584768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145584768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696213.739610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145584896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696213.739631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183645c00 x1874531145584896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145584896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145584896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696213.739681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145585024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696213.739699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183647480 x1874531145585024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145585024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145585024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696213.739746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145585152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696213.739763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183646680 x1874531145585152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145585152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145585152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696213.739817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145585280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696213.739835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696213.739846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696213/real 1787696213] req@ffff8f5183644000 x1874531145585280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696213.739859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145585280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696213.739868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696213.739872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696213.739876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145585280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696218.858762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145585408:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696218.858894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696218.858937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696218.858970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696218.858996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696218.859029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696218.859073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696218.859103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696218.859119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696221.930776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f518738aa00 x1874531145586432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.930814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145586432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.930833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.930842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.930847:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696221.930855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f518738ad80 x1874531145586176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.930869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145586176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.930877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.930880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.930884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696221.930891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f5187389500 x1874531145586304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.930902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145586304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.930910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.930913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.930917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696221.930923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f518738b100 x1874531145585664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.930935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145585664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.930944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.930948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.930951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696221.930959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f5187388380 x1874531145585792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.930972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145585792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.930981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.930985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.930989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696221.930996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f518738a300 x1874531145585920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.931007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145585920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.931016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.931019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.931022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696221.931030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f5187389f80 x1874531145586048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.931042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145586048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.931050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.931054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.931057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145586048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696221.931063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f5187389c00 x1874531145585408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.931076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145585408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.931083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.931086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.931090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145585408:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696221.931096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696218/real 1787696221] req@ffff8f5187389880 x1874531145585536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696221.931109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145585536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696221.931117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696221.931121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696221.931124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145585536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696223.978624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145586560:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696223.978702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.978774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f5187389180 x1874531145586560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.978818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145586560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.978843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.978853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.978860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145586560:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696223.978893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145586688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696223.978942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.978968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f518738b800 x1874531145586688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.978986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145586688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145586688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696223.979041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145586816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696223.979075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f518738bb80 x1874531145586816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145586816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145586816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696223.979140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145586944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696223.979165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f518738b480 x1874531145586944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145586944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145586944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696223.979215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145587072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696223.979238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f5187388000 x1874531145587072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145587072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145587072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696223.979289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696223.979342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f5187388a80 x1874531145587200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145587200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696223.979446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696223.979470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f518738a680 x1874531145587328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145587328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696223.979530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696223.979551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f5187388e00 x1874531145587456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145587456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696223.979595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696223.979616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696223.979628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696223/real 1787696223] req@ffff8f5187389180 x1874531145587584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696223.979641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145587584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696223.979650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696223.979653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696223.979657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145587584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696229.098575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145587712:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696229.098644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.098714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f5187389880 x1874531145587712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.098750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145587712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.098772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.098782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.098791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145587712:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696229.098807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145587840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696229.098838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.098854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f5187389c00 x1874531145587840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.098867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145587840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.098877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.098881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.098885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145587840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696229.098910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145587968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696229.098932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.098946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f5187389f80 x1874531145587968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.098959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145587968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.098967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.098970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.098973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145587968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696229.098980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145588096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696229.099016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.099029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f518738a300 x1874531145588096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.099042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145588096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.099050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.099053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.099056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145588096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696229.099063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145588224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696229.099081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.099094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f5187388380 x1874531145588224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.099106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145588224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.099115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.099118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.099122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145588224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696229.099128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696229.099147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.099159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f518738b100 x1874531145588352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.099172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145588352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.099180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.099184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.099188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696229.099194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696229.099211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.099259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f5187389500 x1874531145588480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.099272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145588480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.099280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.099283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.099287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696229.099293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696229.099314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.099337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f518738ad80 x1874531145588608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.099349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145588608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.099383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.099387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.099391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696229.099397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696229.099417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696229.099429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696229/real 1787696229] req@ffff8f518738aa00 x1874531145588736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696229.099440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145588736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696229.099448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696229.099451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696229.099454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145588736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696234.218580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145588864:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696234.218656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.218725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f518738aa00 x1874531145588864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.218756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145588864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.218772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.218779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.218786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145588864:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696234.218811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145588992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696234.218841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.218860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f518738ad80 x1874531145588992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.218898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145588992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.218917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.218925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.218932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145588992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696234.218940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145589120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696234.218961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.218979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f5187389500 x1874531145589120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.218998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145589120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145589120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696234.219055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145589248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696234.219078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.219090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f518738b100 x1874531145589248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.219103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145589248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145589248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696234.219123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145589376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696234.219175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.219187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f5187388380 x1874531145589376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.219199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145589376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145589376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696234.219233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696234.219253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.219264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f518738a300 x1874531145589504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.219276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145589504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696234.219298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696234.219313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.219324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f5187389f80 x1874531145589632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.219336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145589632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696234.219381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696234.219399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.219410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f5187389c00 x1874531145589760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.219423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145589760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696234.219444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696234.219459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696234.219468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696234/real 1787696234] req@ffff8f5187389880 x1874531145589888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696234.219479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145589888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696234.219488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696234.219492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696234.219495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145589888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696239.338686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145590016:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696239.338788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.338871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f5187389180 x1874531145590016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.338896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145590016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.338918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.338930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.338942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145590016:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696239.338954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696239.339002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f5187388e00 x1874531145590144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145590144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696239.339093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696239.339118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f518738a680 x1874531145590272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145590272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696239.339177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696239.339197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f5187388a80 x1874531145590400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145590400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696239.339249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696239.339267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f5187388000 x1874531145590528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145590528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145590528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696239.339309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145590656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696239.339326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f518738b480 x1874531145590656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145590656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145590656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696239.339405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145590784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696239.339437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f518738bb80 x1874531145590784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145590784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145590784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696239.339496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145590912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696239.339518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f518738b800 x1874531145590912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145590912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145590912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696239.339563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145591040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696239.339577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696239.339584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696239/real 1787696239] req@ffff8f5187389880 x1874531145591040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696239.339592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145591040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696239.339597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696239.339599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696239.339601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145591040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696244.458632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145591168:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696244.458695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.458749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f5187389c00 x1874531145591168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.458783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145591168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.458805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.458815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.458823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145591168:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696244.458834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696244.458868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.458887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f5187389f80 x1874531145591296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.458905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145591296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.458927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.458938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.458947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696244.458989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696244.459020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f518738a300 x1874531145591424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145591424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696244.459076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696244.459096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f5187388380 x1874531145591552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145591552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696244.459136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696244.459153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f518738b100 x1874531145591680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145591680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145591680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696244.459193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145591808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696244.459209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f5187389500 x1874531145591808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145591808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145591808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696244.459256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145591936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696244.459281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f518738ad80 x1874531145591936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145591936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145591936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696244.459321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145592064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696244.459346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f518738aa00 x1874531145592064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145592064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145592064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696244.459426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145592192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696244.459444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696244.459455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696244/real 1787696244] req@ffff8f5187389880 x1874531145592192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696244.459467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145592192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696244.459475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696244.459478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696244.459480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145592192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696249.578601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145592320:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696249.578656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.578694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f518738b800 x1874531145592320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.578720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145592320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.578744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.578749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.578755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145592320:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696249.578777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696249.578803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.578815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f518738bb80 x1874531145592448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.578833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145592448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.578846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.578853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.578859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696249.578876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696249.578900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.578912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f518738b480 x1874531145592576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.578929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145592576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.578937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.578939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.578941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696249.578946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696249.578962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.578972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f5187388000 x1874531145592704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.579000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145592704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.579013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.579018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.579023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696249.579031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696249.579046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.579059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f5187388a80 x1874531145592832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.579078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145592832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.579085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.579088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.579090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145592832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696249.579102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145592960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696249.579114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.579119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f518738a680 x1874531145592960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.579125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145592960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.579130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.579131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.579134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145592960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696249.579137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145593088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696249.579145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.579150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f5187388e00 x1874531145593088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.579156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145593088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.579160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.579162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.579163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145593088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696249.579166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145593216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696249.579174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.579179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f5187389180 x1874531145593216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.579185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145593216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.579189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.579191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.579192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145593216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696249.579195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145593344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696249.579202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696249.579208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696249/real 1787696249] req@ffff8f518738b800 x1874531145593344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696249.579214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145593344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696249.579218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696249.579220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696249.579221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145593344:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696254.698470:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696254.698480:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698488:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698496:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698501:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696254.698533:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145593472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698556:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696254.698560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698564:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698566:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698573:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696254.698580:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145593600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698588:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696254.698591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698597:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698601:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696254.698607:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145593728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698615:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696254.698616:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698619:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698621:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698625:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698629:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696254.698648:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145593856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698662:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696254.698668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698677:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698682:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698687:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696254.698706:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145593984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696254.698717:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698719:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698723:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698730:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696254.698748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145594112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696254.698761:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698764:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698766:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698769:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698772:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696254.698790:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145594240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698800:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698801:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696254.698802:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698806:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698812:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698818:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698826:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696254.698845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145594368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696254.698854:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696254.698856:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696254.698858:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696254.698864:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4154 00000100:00080000:2.0:1787696254.698874:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696254.698888:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696254.698898:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696254.698910:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145594496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696264.938699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145595776:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696264.938786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.938847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183647100 x1874531145595776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.938903:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145595776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.938915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.938929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.938946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145595776:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696264.938970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145595904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696264.939015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183646d80 x1874531145595904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145595904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145595904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696264.939116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145596032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696264.939149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183645180 x1874531145596032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145596032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145596032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696264.939215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145596160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696264.939236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183644380 x1874531145596160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145596160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145596160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696264.939279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145596288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696264.939308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183644700 x1874531145596288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145596288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145596288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696264.939410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696264.939438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183647800 x1874531145596416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145596416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696264.939490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696264.939511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183644000 x1874531145596544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145596544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696264.939558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696264.939578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183646680 x1874531145596672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145596672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696264.939627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696264.939642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696264.939650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696264/real 1787696264] req@ffff8f5183647480 x1874531145596800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696264.939658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145596800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696264.939664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696264.939666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696264.939668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145596800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696270.058659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145596928:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696270.058718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.058770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f5187389c00 x1874531145596928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.058803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145596928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.058824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.058829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.058835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145596928:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696270.058859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696270.058885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.058908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f5187389f80 x1874531145597056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.058927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145597056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.058933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.058935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.058937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696270.058944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696270.058962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.058973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f518738a300 x1874531145597184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.058995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145597184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696270.059036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696270.059067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.059086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f5187388380 x1874531145597312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.059103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145597312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696270.059144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696270.059171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.059184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f518738b100 x1874531145597440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.059197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145597440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145597440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696270.059236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696270.059251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.059259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f5187389500 x1874531145597568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.059267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145597568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696270.059281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696270.059290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.059296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f518738ad80 x1874531145597696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.059303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145597696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696270.059323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696270.059342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.059385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f518738aa00 x1874531145597824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.059400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145597824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696270.059427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696270.059443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696270.059451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696270/real 1787696270] req@ffff8f5187389880 x1874531145597952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696270.059459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145597952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696270.059464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696270.059466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696270.059468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145597952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696275.178604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145598080:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696275.178668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.178721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f518738b800 x1874531145598080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.178750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145598080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.178769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.178776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.178785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145598080:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696275.178796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696275.178821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.178836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f5187389180 x1874531145598208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.178860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145598208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.178877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.178883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.178887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696275.178912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696275.178927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.178934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f5187388e00 x1874531145598336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.178941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145598336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.178947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.178948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.178950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696275.178954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696275.178964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.178970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f518738a680 x1874531145598464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.178978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145598464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.178983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.178984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.178986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696275.178989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696275.178999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.179005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f5187388a80 x1874531145598592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.179012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145598592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.179016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.179018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.179020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145598592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696275.179023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145598720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696275.179033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.179038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f5187388000 x1874531145598720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.179045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145598720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.179055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.179057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.179058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145598720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696275.179061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145598848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696275.179071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.179076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f518738b480 x1874531145598848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.179083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145598848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.179088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.179090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.179092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145598848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696275.179095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145598976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696275.179104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.179110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f518738bb80 x1874531145598976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.179116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145598976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.179121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.179122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.179125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145598976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696275.179129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145599104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696275.179138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696275.179143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696275/real 1787696275] req@ffff8f5187389880 x1874531145599104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696275.179150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145599104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696275.179155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696275.179157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696275.179159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145599104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696280.298707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145599232:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696280.298787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.298869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183645c00 x1874531145599232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.298906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145599232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.298931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.298939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.298947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145599232:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696280.298973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696280.298995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183646a00 x1874531145599360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145599360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696280.299030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696280.299042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183644e00 x1874531145599488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145599488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696280.299069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696280.299080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183647b80 x1874531145599616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145599616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696280.299120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696280.299132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183645500 x1874531145599744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645500 x1874531145599744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145599744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696280.299170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145599872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696280.299192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183647480 x1874531145599872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145599872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145599872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696280.299228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145600000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696280.299242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183646680 x1874531145600000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145600000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145600000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696280.299267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145600128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696280.299276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183644000 x1874531145600128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145600128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145600128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696280.299299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145600256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696280.299309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696280.299316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696280/real 1787696280] req@ffff8f5183647800 x1874531145600256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696280.299328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145600256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696280.299390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696280.299393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696280.299395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145600256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696285.418641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145600384:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696285.418795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696285.418849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696285.418876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696285.418908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696285.418932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696285.418956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696285.418980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696285.418999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696288.490896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183647480 x1874531145601408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.490942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145601408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.490958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.490965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.490971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696288.490981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183646680 x1874531145601280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145601280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696288.491036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183645180 x1874531145600640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145600640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696288.491078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183646d80 x1874531145600768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145600768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696288.491129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183647100 x1874531145600896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145600896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696288.491176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183647800 x1874531145601024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145601024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696288.491248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183644000 x1874531145601152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145601152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145601152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696288.491281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183644700 x1874531145600384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145600384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145600384:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696288.491314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696285/real 1787696288] req@ffff8f5183644380 x1874531145600512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696288.491324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145600512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696288.491332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696288.491335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696288.491339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145600512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696290.538746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145601536:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696290.538855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.538924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f518738aa00 x1874531145601536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.538965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145601536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.538990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145601536:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696290.539037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145601664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696290.539071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f518738ad80 x1874531145601664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145601664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145601664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696290.539202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145601792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696290.539229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f5187389500 x1874531145601792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145601792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145601792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696290.539292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145601920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696290.539316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f518738b100 x1874531145601920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145601920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145601920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696290.539402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145602048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696290.539426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f5187388380 x1874531145602048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145602048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145602048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696290.539471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696290.539491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f518738a300 x1874531145602176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145602176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696290.539539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696290.539561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f5187389f80 x1874531145602304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145602304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696290.539617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696290.539641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f5187389c00 x1874531145602432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145602432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696290.539689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696290.539711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696290.539725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696290/real 1787696290] req@ffff8f5187389880 x1874531145602560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696290.539739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145602560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696290.539748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696290.539752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696290.539756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145602560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696295.658603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145602688:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696295.658691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.658758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f518738bb80 x1874531145602688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.658787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145602688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.658812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.658818:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.658826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145602688:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696295.658840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145602816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696295.658869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.658886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f518738b480 x1874531145602816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.658900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145602816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.658910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.658914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.658917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145602816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696295.658949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145602944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696295.658973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.658986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f5187388000 x1874531145602944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.658995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145602944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145602944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696295.659011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145603072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696295.659027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.659034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f5187388a80 x1874531145603072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.659042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145603072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145603072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696295.659056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145603200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696295.659066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.659092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f518738a680 x1874531145603200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.659100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145603200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145603200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696295.659114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696295.659129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.659136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f5187388e00 x1874531145603328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.659144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145603328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696295.659158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696295.659169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.659175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f5187389180 x1874531145603456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.659183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145603456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696295.659195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696295.659204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.659212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f518738b800 x1874531145603584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.659222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145603584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696295.659251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696295.659274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696295.659285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696295/real 1787696295] req@ffff8f5187389880 x1874531145603712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696295.659293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145603712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696295.659299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696295.659301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696295.659303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145603712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696300.778709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145603840:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696300.778807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.778890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f5187389c00 x1874531145603840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.778944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145603840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.778966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.778977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145603840:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696300.779039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145603968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696300.779079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f5187389f80 x1874531145603968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145603968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145603968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696300.779220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145604096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696300.779255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f518738a300 x1874531145604096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145604096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145604096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696300.779317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145604224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696300.779337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f5187388380 x1874531145604224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145604224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145604224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696300.779419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145604352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696300.779439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f518738b100 x1874531145604352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145604352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145604352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696300.779484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696300.779506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f5187389500 x1874531145604480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145604480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696300.779558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696300.779576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f518738ad80 x1874531145604608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145604608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696300.779617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696300.779635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f518738aa00 x1874531145604736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145604736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696300.779693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696300.779712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696300.779722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696300/real 1787696300] req@ffff8f5187389f80 x1874531145604864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696300.779734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145604864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696300.779742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696300.779745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696300.779749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145604864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696305.898581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145604992:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696305.898678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.898739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f5187389c00 x1874531145604992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.898781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145604992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.898803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.898812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.898826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145604992:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696305.898866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696305.898913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.898938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f5187389880 x1874531145605120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.898971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145605120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696305.899063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696305.899117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f518738b800 x1874531145605248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145605248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696305.899181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696305.899204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f5187389180 x1874531145605376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145605376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696305.899247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696305.899284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f5187388e00 x1874531145605504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145605504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145605504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696305.899327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145605632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696305.899341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f518738a680 x1874531145605632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145605632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145605632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696305.899407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145605760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696305.899420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f5187388a80 x1874531145605760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145605760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145605760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696305.899448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145605888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696305.899459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f5187388000 x1874531145605888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145605888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145605888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696305.899484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145606016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696305.899494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696305.899500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696305/real 1787696305] req@ffff8f518738b480 x1874531145606016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696305.899507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145606016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696305.899513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696305.899514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696305.899516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145606016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696311.018559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145606144:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696311.018621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.018678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f518738bb80 x1874531145606144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.018704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145606144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.018718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.018731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.018735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145606144:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696311.018744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696311.018772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.018787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f518738b480 x1874531145606272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.018802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145606272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.018858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.018870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.018876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696311.018908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696311.018937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.018947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f5187388000 x1874531145606400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.018956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145606400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.018962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.018964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.018966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696311.018970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696311.018983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.018989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f5187388a80 x1874531145606528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.018997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145606528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.019001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.019003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.019006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696311.019010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696311.019019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.019025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f518738a680 x1874531145606656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.019032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145606656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.019037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.019039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.019041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145606656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696311.019044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145606784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696311.019054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.019060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f5187388e00 x1874531145606784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.019071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145606784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.019081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.019085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.019088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145606784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696311.019095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145606912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696311.019115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.019127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f5187389180 x1874531145606912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.019135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145606912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.019141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.019143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.019145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145606912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696311.019148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145607040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696311.019161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.019167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f518738b800 x1874531145607040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.019174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145607040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.019179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.019181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.019183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145607040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696311.019186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145607168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696311.019195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696311.019200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696311/real 1787696311] req@ffff8f5187389880 x1874531145607168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696311.019207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389880 x1874531145607168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696311.019213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696311.019215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696311.019217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145607168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696316.138658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145607296:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696316.138749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.138809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f5187389c00 x1874531145607296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.138843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145607296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.138854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.138863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.138870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145607296:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696316.138896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696316.138932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.138952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f5187389f80 x1874531145607424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.138987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145607424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696316.139034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696316.139058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f518738aa00 x1874531145607552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145607552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696316.139133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696316.139152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f518738ad80 x1874531145607680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145607680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696316.139202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696316.139213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f5187389500 x1874531145607808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145607808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145607808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696316.139249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145607936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696316.139259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f518738b100 x1874531145607936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145607936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145607936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696316.139285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145608064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696316.139295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f5187388380 x1874531145608064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145608064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145608064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696316.139331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145608192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696316.139341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f518738a300 x1874531145608192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145608192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145608192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696316.139386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145608320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696316.139398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696316.139406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696316/real 1787696316] req@ffff8f5187389c00 x1874531145608320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696316.139413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145608320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696316.139418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696316.139420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696316.139423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145608320:192.168.201.125@tcp:8:kworker.0 00010000:00010000:2.0:1787696317.034513:0:55697:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1787696317.034569:0:55697:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 5/5 locks to regular blp list, 1 blwis in pool 00000100:00080000:2.0:1787696317.034843:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8f5187389880 x1874531145608448/t0(0) o103->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.258699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145608704:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696321.258797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.258856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f518738b800 x1874531145608704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.258883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145608704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.258903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.258909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.258927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145608704:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696321.258966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145608832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696321.258998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f5187389180 x1874531145608832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145608832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145608832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696321.259048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145608960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696321.259068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f5187388e00 x1874531145608960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145608960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145608960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696321.259115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145609088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696321.259135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f518738a680 x1874531145609088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145609088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145609088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696321.259180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145609216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696321.259199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f5187388a80 x1874531145609216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145609216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145609216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696321.259244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696321.259263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f5187388000 x1874531145609344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145609344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696321.259308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696321.259328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f518738b480 x1874531145609472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145609472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696321.259397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696321.259418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f518738bb80 x1874531145609600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145609600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696321.259489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696321.259510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696321.259522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696321/real 1787696321] req@ffff8f518738b800 x1874531145609728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696321.259534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145609728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696321.259542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696321.259545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696321.259549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145609728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696326.378628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145609856:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696326.378729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.378801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f5187389c00 x1874531145609856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.378870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145609856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.378896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.378905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.378910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145609856:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696326.378924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145609984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696326.378949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.378973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f518738a300 x1874531145609984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.378986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145609984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.378994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.378997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145609984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696326.379037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145610112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696326.379057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f5187388380 x1874531145610112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145610112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145610112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696326.379101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145610240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696326.379120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f518738b100 x1874531145610240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145610240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145610240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696326.379171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145610368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696326.379188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f5187389500 x1874531145610368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145610368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145610368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696326.379245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696326.379265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f518738ad80 x1874531145610496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145610496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696326.379315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696326.379333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f518738aa00 x1874531145610624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145610624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696326.379427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696326.379446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f5187389f80 x1874531145610752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145610752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696326.379489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696326.379505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696326.379514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696326/real 1787696326] req@ffff8f518738b800 x1874531145610880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696326.379526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145610880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696326.379534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696326.379537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696326.379552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145610880:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696331.498465:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498474:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696331.498478:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498487:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498495:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498502:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696331.498539:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145611008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498565:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498568:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696331.498570:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498574:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498576:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498580:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498588:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696331.498602:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145611136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498620:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498625:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696331.498627:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498632:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498636:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498641:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696331.498661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145611264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498684:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498688:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696331.498690:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498695:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498712:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696331.498734:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145611392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498765:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696331.498767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498775:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498778:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498782:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498786:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696331.498799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145611520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498809:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696331.498813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498820:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498825:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498829:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498834:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696331.498845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145611648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498873:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498876:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696331.498878:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498883:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498888:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498894:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696331.498912:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145611776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.498934:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.498938:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696331.498941:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.498947:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.498951:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.498959:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.498976:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696331.498991:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145611904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696331.499017:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696331.499032:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696331.499039:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696331.499047:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4231 00000100:00080000:2.0:1787696331.499053:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696331.499060:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696331.499069:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696331.499087:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145612032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218522:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218533:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696362.218537:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218548:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218560:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218568:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696362.218611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145617920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218643:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218647:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696362.218650:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218656:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218660:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218666:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.218684:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696362.218700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145618048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218718:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218722:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696362.218723:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218728:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218731:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218734:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.218738:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696362.218752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145618176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696362.218763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218766:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218768:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.218774:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696362.218785:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145618304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218802:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696362.218808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218812:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218817:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218820:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.218823:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696362.218831:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145618432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218838:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218851:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696362.218852:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218859:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218863:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218866:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.218871:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696362.218888:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145618560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.218913:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.218945:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696362.218947:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.218952:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.218969:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.218972:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.218982:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696362.219003:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145618688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.219013:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.219019:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696362.219023:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.219035:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.219039:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.219050:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.219055:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696362.219065:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145618816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696362.219073:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696362.219077:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696362.219081:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696362.219089:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4261 00000100:00080000:2.0:1787696362.219092:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696362.219095:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696362.219099:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696362.219107:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145618944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696367.338482:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338494:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338507:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338516:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696367.338563:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145619072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338589:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696367.338590:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338594:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338608:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696367.338618:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145619200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338625:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338628:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696367.338629:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338632:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338634:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338636:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338644:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696367.338655:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145619328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696367.338672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338675:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338677:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338680:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338683:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696367.338692:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145619456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338699:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696367.338703:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338705:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338710:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338713:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696367.338720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145619584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696367.338729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338732:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338737:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338744:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696367.338756:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145619712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338764:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696367.338768:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338770:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338772:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338775:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338778:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696367.338786:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145619840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338798:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696367.338808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338813:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338817:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338820:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338825:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696367.338854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145619968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696367.338889:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696367.338892:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696367.338894:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696367.338898:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4266 00000100:00080000:2.0:1787696367.338903:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696367.338906:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696367.338911:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696367.338924:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145620096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696377.578674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145621376:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696377.578754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.578823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183647b80 x1874531145621376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.578859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145621376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.578879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.578890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.578896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145621376:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696377.578906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696377.578946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.578961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183644e00 x1874531145621504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.578975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145621504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.578983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.578986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.578990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696377.579016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696377.579038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183646a00 x1874531145621632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145621632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696377.579085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696377.579104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183645c00 x1874531145621760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145621760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696377.579153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696377.579171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183644380 x1874531145621888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145621888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145621888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696377.579216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696377.579236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183644700 x1874531145622016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145622016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696377.579280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696377.579303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183644000 x1874531145622144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145622144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696377.579349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696377.579395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183647800 x1874531145622272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145622272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696377.579439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696377.579456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696377.579467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696377/real 1787696377] req@ffff8f5183647100 x1874531145622400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696377.579479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145622400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696377.579487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696377.579490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696377.579494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145622400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696382.698663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145622528:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696382.698732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.698787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f518738b480 x1874531145622528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.698809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145622528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.698825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.698834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.698845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145622528:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696382.698888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145622656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696382.698919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.698937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f5187388000 x1874531145622656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.698963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145622656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.698991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.698995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145622656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696382.699008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145622784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696382.699029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f5187388a80 x1874531145622784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145622784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145622784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696382.699058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145622912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696382.699069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f518738a680 x1874531145622912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145622912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145622912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696382.699106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145623040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696382.699126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f5187388e00 x1874531145623040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145623040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145623040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696382.699160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696382.699174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f5187389180 x1874531145623168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145623168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696382.699200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696382.699210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f5187389f80 x1874531145623296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145623296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696382.699247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696382.699257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f518738b480 x1874531145623424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145623424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696382.699282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696382.699291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696382.699296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696382/real 1787696382] req@ffff8f518738b800 x1874531145623552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696382.699303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145623552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696382.699309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696382.699311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696382.699314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145623552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787696387.818552:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145623680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696387.818575:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696387.818578:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696387.818580:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.818583:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818585:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.818588:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818595:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696387.818603:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145623808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696387.818611:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696387.818613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696387.818614:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.818617:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818619:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.818622:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818626:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696387.818631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145623680:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696387.818641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145623936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696387.818654:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696387.818658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696387.818660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.818664:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818680:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.818685:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787696387.818689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696387.818704:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145624064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696387.818711:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696387.818713:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696387.818715:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.818718:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818721:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.818724:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696387.818730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f518738bb80 x1874531145623680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696387.818745:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145624192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696387.818749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145623680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.818773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696387.818774:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696387.818781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696387.818782:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787696387.818787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145623680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696387.818788:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696387.818794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145623808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696387.818797:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818804:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.818809:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787696387.818814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696387.818824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f518738aa00 x1874531145623808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696387.818833:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145624320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696387.818837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145623808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.818861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.818867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696387.818868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696387.818874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145623808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696387.818874:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696387.818878:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.818884:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818887:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.818890:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696387.818900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145623936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696387.818905:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145624448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696387.818918:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787696387.818923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696387.818925:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696387.818929:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.818935:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00080000:3.0:1787696387.818941:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696387.818944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f518738ad80 x1874531145623936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696387.818948:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696387.818955:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696387.818967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145623936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696387.818969:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145624576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696387.818995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696387.818997:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696387.819001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696387.819003:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787696387.819007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145623936:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696387.819008:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696387.819019:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4287 00000100:00100000:2.0:1787696387.819020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145624064:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787696387.819027:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696387.819034:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696387.819040:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787696387.819045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696387.819063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f5187389500 x1874531145624064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696387.819064:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145624704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696387.819083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145624064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.819091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.819094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696387.819097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145624064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696387.819101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145624192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696387.819121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696387.819129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f518738b100 x1874531145624192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696387.819137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145624192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.819142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.819144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696387.819146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145624192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696387.819149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696387.819161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696387.819166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f5187388380 x1874531145624320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696387.819174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145624320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.819180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.819181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696387.819183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696387.819194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696387.819204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696387.819210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f518738a300 x1874531145624448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696387.819217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145624448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.819227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.819229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696387.819231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696387.819234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696387.819244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696387.819249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f5187389c00 x1874531145624576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696387.819256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145624576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.819261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.819262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696387.819264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696387.819267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696387.819277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696387.819282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696387/real 1787696387] req@ffff8f518738aa00 x1874531145624704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696387.819289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145624704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696387.819294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696387.819295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696387.819298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145624704:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696388.714516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696389.738618:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696392.810550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696392.938484:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938493:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696392.938497:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938505:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938514:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938520:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696392.938552:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145624832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938574:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938577:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696392.938578:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938581:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938584:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938587:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938593:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696392.938602:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145624960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938612:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696392.938613:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938618:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00100000:2.0:1787696392.938620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145624832:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696392.938621:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938627:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938631:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696392.938641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145625088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938649:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696392.938651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938653:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938656:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938658:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696392.938668:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145625216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938674:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696392.938678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938680:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938684:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938686:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938688:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787696392.938696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696392.938697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145625344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938708:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696392.938709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938713:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938715:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938718:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938721:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696392.938727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145625472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938735:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696392.938736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938739:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938742:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696392.938747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f518738bb80 x1874531145624832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696392.938747:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938750:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696392.938760:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145625600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696392.938769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145624832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696392.938778:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696392.938785:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696392.938786:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696392.938787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696392.938791:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:2.0:1787696392.938795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696392.938795:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696392.938801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145624832:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696392.938802:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938807:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696392.938823:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145625728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696392.938831:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696392.938833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145624960:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696392.938834:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696392.938838:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696392.938845:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4292 00000100:00080000:3.0:1787696392.938850:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696392.938854:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696392.938859:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696392.938872:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145625856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696392.938881:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.938901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f518738b800 x1874531145624960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.938915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145624960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.938923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.938925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.938928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145624960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696392.938934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145625088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696392.938949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.938957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f518738b480 x1874531145625088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.938964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145625088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.938969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.938971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.938973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145625088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696392.938977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145625216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696392.938987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.938993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f5187389f80 x1874531145625216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.939000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145625216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.939005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.939007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.939009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145625216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696392.939012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145625344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696392.939021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.939035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f5187389180 x1874531145625344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.939042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145625344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.939048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.939050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.939052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145625344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696392.939056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696392.939066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.939072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f5187388e00 x1874531145625472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.939079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145625472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.939085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.939087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.939088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696392.939092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696392.939102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.939108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f518738a680 x1874531145625600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.939115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145625600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.939119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.939121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.939124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696392.939137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696392.939148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.939154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f5187388a80 x1874531145625728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.939161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145625728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.939166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.939168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.939170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696392.939173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696392.939182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696392.939188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696392/real 1787696392] req@ffff8f5187388000 x1874531145625856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696392.939195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145625856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696392.939200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696392.939202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696392.939204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145625856:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696393.834606:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696396.906518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696397.930633:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696398.058435:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.058450:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696398.058457:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.058469:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:3.0:1787696398.058482:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696398.058490:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696398.058533:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145625984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696398.058561:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.058565:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696398.058567:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.058572:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:3.0:1787696398.058577:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696398.058583:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696398.058595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696398.058600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145625984:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696398.058616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145626112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696398.058629:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.058633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696398.058636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.058641:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:3.0:1787696398.058645:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696398.058650:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696398.058656:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696398.058671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145626240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696398.058689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696398.058695:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.058700:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696398.058703:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.058708:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:3.0:1787696398.058713:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696398.058718:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696398.058724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696398.058738:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145626368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696398.058749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f5187388000 x1874531145625984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696398.058752:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.058764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696398.058768:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.058774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00100000:2.0:1787696398.058777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145625984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696398.058780:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696398.058789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696398.058790:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787696398.058795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.058800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145625984:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787696398.058801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696398.058818:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145626496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696398.058830:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696398.058834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696398.058836:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696398.058842:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.058849:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:3.0:1787696398.058864:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787696398.058865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696398.058874:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696398.058882:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696398.058887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f5187388a80 x1874531145626112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696398.058901:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145626624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696398.058919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145626112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696398.058920:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.058930:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696398.058936:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696398.058944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696398.058945:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:2.0:1787696398.058953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696398.058955:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696398.058960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696398.058966:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696398.058975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626240:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787696398.058976:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696398.059000:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145626752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696398.059012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696398.059014:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.059020:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696398.059022:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696398.059029:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00100000:2.0:1787696398.059031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f518738a680 x1874531145626240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696398.059035:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696398.059045:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696398.059053:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696398.059068:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145626880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696398.059072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145626240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696398.059106:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696398.059116:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696398.059120:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696398.059124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696398.059129:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4297 00000100:00080000:2.0:1787696398.059132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626240:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696398.059139:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696398.059150:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696398.059151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626368:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787696398.059162:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787696398.059181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696398.059181:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145627008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696398.059193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f5187388e00 x1874531145626368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696398.059206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145626368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696398.059215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696398.059220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696398.059230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696398.059251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696398.059264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f5187389180 x1874531145626496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696398.059276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145626496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696398.059282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696398.059284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145626496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696398.059301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145626624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696398.059332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696398.059341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f5187389f80 x1874531145626624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696398.059349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145626624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696398.059376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696398.059379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145626624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696398.059387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145626752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696398.059402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696398.059411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f518738b480 x1874531145626752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696398.059420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145626752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696398.059425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696398.059427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145626752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696398.059432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145626880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696398.059443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696398.059449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f518738b800 x1874531145626880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696398.059456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145626880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696398.059461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696398.059463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145626880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696398.059468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145627008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696398.059478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696398.059484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696398/real 1787696398] req@ffff8f518738bb80 x1874531145627008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696398.059491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145627008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696398.059496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696398.059497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696398.059499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145627008:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696401.002502:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696402.026493:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696403.178528:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178541:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696403.178546:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178558:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178571:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178579:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696403.178636:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145627136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178669:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178672:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696403.178674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178678:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178681:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178684:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696403.178703:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145627264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178712:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696403.178716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178719:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178721:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178723:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178726:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696403.178735:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145627392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178743:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696403.178745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145627136:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696403.178747:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696403.178750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178756:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178759:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178761:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178764:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696403.178773:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145627520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178786:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178788:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696403.178790:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178793:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178811:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178814:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178817:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696403.178825:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145627648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178832:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787696403.178838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696403.178838:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696403.178842:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178847:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178849:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178852:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178855:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696403.178864:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145627776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178871:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178873:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696403.178875:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178877:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178881:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178883:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178887:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696403.178895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145627904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178900:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178903:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696403.178904:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696403.178911:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00100000:2.0:1787696403.178913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f518738aa00 x1874531145627136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696403.178914:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178919:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178926:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696403.178938:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145628032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696403.178945:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696403.178948:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696403.178950:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696403.178953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145627136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696403.178955:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4302 00000100:00080000:3.0:1787696403.178960:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696403.178966:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696403.178973:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787696403.178983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696403.178984:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145628160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696403.178994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145627136:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696403.179017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696403.179044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f5187389c00 x1874531145627264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145627264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696403.179103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696403.179116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f518738a300 x1874531145627392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145627392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696403.179146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696403.179158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f5187388380 x1874531145627520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145627520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696403.179184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696403.179195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f518738b100 x1874531145627648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145627648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145627648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696403.179239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145627776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696403.179255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f5187389500 x1874531145627776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145627776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145627776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696403.179287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145627904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696403.179298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f518738ad80 x1874531145627904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145627904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145627904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696403.179325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145628032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696403.179335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f518738bb80 x1874531145628032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145628032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145628032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696403.179399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145628160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696403.179414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696403.179422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696403/real 1787696403] req@ffff8f518738b800 x1874531145628160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696403.179430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145628160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696403.179435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696403.179438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696403.179440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145628160:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696405.098532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696406.122550:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696408.298472:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298486:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696408.298492:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298507:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298521:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.298530:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696408.298583:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145628288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.298619:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298625:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696408.298627:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298640:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298644:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.298651:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.298661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696408.298675:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145628416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.298692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696408.298699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298705:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298711:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.298715:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.298731:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696408.298748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145628544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.298759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696408.298766:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298771:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298775:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.298784:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.298794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696408.298808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145628672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.298820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298824:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696408.298826:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298830:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298835:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.298840:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.298846:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696408.298863:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145628800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.298884:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298890:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696408.298896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298905:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298910:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.298916:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.298922:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696408.298940:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145628928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.298954:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.298962:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696408.298978:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.298988:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.298997:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.299006:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.299018:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696408.299039:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145629056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.299055:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.299066:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696408.299070:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.299080:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.299089:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.299100:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.299112:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696408.299144:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145629184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696408.299159:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696408.299163:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696408.299166:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696408.299172:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4307 00000100:00080000:2.0:1787696408.299180:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696408.299189:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696408.299200:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696408.299223:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145629312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696409.194536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696410.218630:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696413.290619:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696413.418519:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418531:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696413.418535:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418547:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418568:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418575:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696413.418621:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145629440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418650:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696413.418654:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418658:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418661:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418664:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418673:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696413.418683:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145629568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418694:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696413.418697:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418703:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418708:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418713:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696413.418726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145629696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418732:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696413.418736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418739:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418741:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418743:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418747:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696413.418754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145629824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696413.418763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418766:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418769:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418773:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696413.418781:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145629952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418787:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696413.418794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418801:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418803:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418806:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418810:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696413.418819:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145630080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418828:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696413.418829:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418831:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418834:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418836:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418839:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696413.418847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145630208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418853:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418856:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696413.418857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418865:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418869:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418875:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418883:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696413.418897:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145630336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696413.418908:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696413.418914:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696413.418919:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696413.418929:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4313 00000100:00080000:3.0:1787696413.418937:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696413.418942:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696413.418950:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696413.418965:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145630464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696414.314594:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696417.386602:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696418.410545:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696421.482566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696421.610634:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696421.610684:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696421.610708:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610772:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610792:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610804:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610818:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610836:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610875:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610891:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696421.610945:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00010000:00010000:3.0:1787696422.506525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696425.578597:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696426.602555:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696429.674539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696430.698545:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696433.770542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696433.898494:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.898504:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696433.898509:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.898519:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.898530:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.898538:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696433.898579:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145634048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.898612:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.898616:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696433.898619:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.898624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.898628:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.898632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.898642:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696433.898661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145634176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.898672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.898676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696433.898679:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.898684:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.898705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.898714:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.898723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696433.898747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145634304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.898761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.898765:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696433.898767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.898773:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.898780:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.898785:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.898790:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696433.898804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145634432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.898837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.898843:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696433.898845:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.898853:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.898859:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.898866:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.898872:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696433.898906:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145634560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.898931:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.898935:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696433.898938:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.898947:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.898958:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.898967:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.898975:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696433.898996:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145634688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.899021:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.899029:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696433.899034:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.899040:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.899045:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.899050:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.899055:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696433.899071:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145634816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.899087:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.899100:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696433.899106:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.899121:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.899127:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.899136:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.899144:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696433.899161:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145634944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696433.899193:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696433.899199:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696433.899202:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696433.899208:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4333 00000100:00080000:3.0:1787696433.899214:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696433.899219:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696433.899224:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696433.899242:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145635072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696434.794531:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696437.866539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696438.890545:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696439.018477:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018488:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696439.018492:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018503:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018514:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018522:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696439.018560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145635200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018589:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696439.018590:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018594:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018611:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018616:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018626:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696439.018641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145635328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018652:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696439.018658:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018662:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018667:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018672:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018678:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696439.018693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145635456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018710:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696439.018712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018715:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018718:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018722:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018726:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696439.018735:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145635584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018742:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018745:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696439.018746:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018751:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018754:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018758:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018761:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696439.018770:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145635712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018777:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018779:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696439.018780:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018783:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018785:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018788:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018793:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696439.018807:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145635840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018824:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696439.018828:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018837:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018856:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018868:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018879:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696439.018896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145635968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018909:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018914:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696439.018916:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.018921:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.018926:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.018931:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.018937:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696439.018952:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145636096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696439.018995:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696439.018999:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696439.019001:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696439.019008:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4338 00000100:00080000:2.0:1787696439.019015:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696439.019021:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696439.019027:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696439.019047:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145636224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696441.962525:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696442.986575:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696444.138569:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138580:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696444.138585:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.138597:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.138610:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.138618:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696444.138668:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145636352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.138703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696444.138711:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.138717:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.138721:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.138726:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.138738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696444.138764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145636480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.138777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138781:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696444.138783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.138788:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.138792:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.138796:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.138801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696444.138814:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145636608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.138827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138833:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696444.138835:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.138840:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.138843:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.138847:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.138852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696444.138863:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145636736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.138873:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138877:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696444.138879:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.138884:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.138887:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.138894:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.138903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696444.138916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145636864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.138926:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138935:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696444.138938:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.138944:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.138950:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.138958:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.138963:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696444.138984:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145636992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.138994:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.138998:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696444.139000:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.139009:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.139015:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.139020:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.139026:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696444.139046:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145637120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.139071:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.139074:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696444.139076:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.139082:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.139086:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.139092:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.139100:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696444.139110:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145637248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696444.139130:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696444.139135:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696444.139138:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696444.139144:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4343 00000100:00080000:3.0:1787696444.139146:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696444.139149:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696444.139153:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696444.139161:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145637376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696446.058537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696447.082543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696449.258473:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258488:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696449.258493:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258507:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258520:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258528:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696449.258576:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145637504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258611:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696449.258616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258620:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258624:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258638:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696449.258651:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145637632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258660:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696449.258666:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258675:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258677:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258680:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696449.258694:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145637760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258704:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696449.258705:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258708:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258709:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258712:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696449.258723:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145637888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696449.258738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258740:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258743:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258751:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696449.258764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145638016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696449.258775:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258781:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258784:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696449.258799:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145638144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258831:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258834:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696449.258836:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258846:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258855:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258862:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696449.258879:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145638272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258902:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258905:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696449.258906:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258912:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258919:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258924:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.258928:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696449.258942:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145638400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696449.258953:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696449.258961:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696449.258967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696449.258975:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4348 00000100:00080000:3.0:1787696449.258983:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696449.258994:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696449.259006:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696449.259025:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145638528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696450.154625:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696451.178579:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696454.250469:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696455.274515:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696458.346559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696459.370562:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696462.442654:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696463.466560:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696464.618485:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696464.618504:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618517:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618529:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618537:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696464.618577:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145640960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618612:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696464.618620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618627:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618633:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618637:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.618650:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696464.618670:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145641088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618683:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618687:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696464.618690:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618695:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618700:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618704:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.618711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696464.618728:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145641216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618738:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696464.618742:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618750:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618755:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.618762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696464.618783:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145641344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618818:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696464.618821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618825:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618828:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618832:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.618839:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696464.618861:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145641472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618870:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618875:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696464.618877:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618884:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618887:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618889:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.618892:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696464.618903:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145641600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618910:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618914:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696464.618915:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618920:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.618927:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.618930:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.618934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696464.618947:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145641728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.618962:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.618972:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696464.618984:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.618997:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.619003:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.619013:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.619021:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696464.619038:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145641856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696464.619047:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696464.619050:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696464.619051:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696464.619056:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4364 00000100:00080000:2.0:1787696464.619061:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696464.619066:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696464.619070:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696464.619080:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145641984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696466.538548:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696467.562561:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696470.634545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696471.658561:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696474.730552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696474.858516:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858530:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696474.858536:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.858549:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.858563:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.858572:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696474.858624:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145643264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.858661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858666:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696474.858668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.858673:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.858677:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.858682:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.858694:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696474.858709:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145643392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.858729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696474.858736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.858742:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.858747:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.858752:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.858759:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696474.858774:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145643520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.858785:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858789:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696474.858791:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.858796:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.858800:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.858807:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.858816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696474.858829:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145643648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.858840:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858844:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696474.858846:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.858850:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.858854:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.858862:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.858868:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696474.858885:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145643776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.858899:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858909:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696474.858915:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.858922:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.858929:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.858939:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.858945:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696474.858958:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145643904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.858994:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.858999:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696474.859002:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.859008:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.859015:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.859023:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.859029:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696474.859052:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145644032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.859091:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.859100:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696474.859105:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.859113:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.859119:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.859125:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.859136:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696474.859156:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145644160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696474.859183:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696474.859193:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696474.859196:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696474.859203:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4374 00000100:00080000:2.0:1787696474.859208:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696474.859216:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696474.859225:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696474.859244:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145644288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696475.754580:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696478.826498:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696479.850602:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696479.978429:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978441:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696479.978446:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978457:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978470:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978477:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696479.978522:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145644416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978551:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978554:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696479.978555:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978559:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978562:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978566:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978575:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696479.978585:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145644544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978593:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978596:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696479.978597:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978600:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978602:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978604:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978607:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696479.978616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145644672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978628:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978631:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696479.978632:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978635:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978637:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978639:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978642:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696479.978652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145644800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978659:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978662:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696479.978663:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978666:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978668:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978670:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696479.978681:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145644928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696479.978690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978693:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978696:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978698:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696479.978712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145645056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978724:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696479.978725:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978728:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978739:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978743:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978749:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696479.978762:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145645184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978775:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978784:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696479.978788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978796:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978800:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978803:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978808:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696479.978822:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145645312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696479.978836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696479.978842:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696479.978844:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696479.978850:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4379 00000100:00080000:2.0:1787696479.978855:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696479.978861:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696479.978867:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696479.978879:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145645440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696482.922537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696483.946582:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696487.018561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696488.042606:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696488.170720:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696488.170812:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696488.170835:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170909:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170927:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170937:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170951:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170961:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170971:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170982:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696488.170996:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00010000:00010000:3.0:1787696491.114580:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696492.138562:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696495.210518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696496.234482:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696499.306587:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696500.330564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696503.402623:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696504.426610:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696507.498525:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696508.522600:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696510.698516:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698527:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696510.698532:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698542:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698554:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698561:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696510.698601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145651328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698633:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698635:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696510.698637:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698641:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698643:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698650:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696510.698673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145651456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698690:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696510.698691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698695:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698698:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698700:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698704:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696510.698715:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145651584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698728:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696510.698732:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698735:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698737:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698739:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698742:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696510.698751:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145651712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698763:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696510.698766:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698770:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698772:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698775:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698778:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696510.698785:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145651840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698791:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698794:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696510.698795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698798:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698802:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698805:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698809:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696510.698821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145651968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696510.698836:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698849:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698856:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698863:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698871:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696510.698880:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145652096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698886:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698888:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696510.698892:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698900:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698904:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698907:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696510.698923:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145652224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696510.698941:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696510.698947:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696510.698949:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696510.698955:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4410 00000100:00080000:2.0:1787696510.698959:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696510.698960:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696510.698963:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696510.698973:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145652352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696511.594474:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696512.618572:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696515.690560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696516.714531:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696519.786483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696520.810525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696523.882537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696524.906505:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696526.058553:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.058569:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696526.058574:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.058588:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.058602:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.058612:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696526.058666:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145654784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.058704:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.058710:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696526.058712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.058719:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.058723:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.058729:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.058743:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696526.058764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145654912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.058781:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.058786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696526.058788:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.058794:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.058799:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.058805:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.058812:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696526.058835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145655040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.058850:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.058854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696526.058857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.058864:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.058869:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.058880:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.058898:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696526.058923:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145655168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.058953:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.058958:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696526.058962:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.058982:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.058989:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.058999:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.059009:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696526.059042:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145655296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.059088:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.059093:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696526.059112:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.059119:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.059125:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.059139:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.059153:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696526.059183:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145655424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.059220:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.059224:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696526.059234:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.059244:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.059256:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.059270:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.059282:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696526.059307:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145655552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.059321:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.059330:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696526.059336:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.059342:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.059350:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.059430:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.059443:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696526.059473:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145655680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696526.059520:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696526.059531:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696526.059540:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696526.059551:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4425 00000100:00080000:3.0:1787696526.059559:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696526.059569:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696526.059576:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696526.059602:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145655808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696527.978596:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696529.002556:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696532.074477:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696533.098492:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696536.170562:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696537.194535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696540.266578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696541.290495:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696544.362559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696545.386567:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696548.458523:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696549.482528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696551.658519:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658530:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696551.658534:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658545:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658557:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658564:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696551.658600:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145660544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658625:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658628:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696551.658629:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658633:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658635:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658639:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696551.658668:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145660672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658680:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696551.658681:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658687:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658693:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658698:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696551.658712:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145660800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658719:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658722:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696551.658723:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658725:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658728:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658731:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658734:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696551.658740:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145660928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696551.658751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658753:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658757:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658759:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696551.658769:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145661056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658775:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696551.658779:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658783:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658785:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658788:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696551.658805:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145661184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658818:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658822:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696551.658824:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658833:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658841:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658850:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696551.658866:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145661312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658878:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658884:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696551.658886:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658891:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658896:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658900:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658906:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696551.658921:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145661440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696551.658932:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696551.658937:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696551.658939:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696551.658960:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4451 00000100:00080000:3.0:1787696551.658967:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696551.658971:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696551.658977:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696551.658995:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145661568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696552.554505:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696553.578534:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696556.650513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696556.778483:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778491:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696556.778494:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778514:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778524:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778530:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696556.778560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145661696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.778593:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778596:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696556.778598:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778602:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778615:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778618:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.778627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696556.778647:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145661824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.778664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778668:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696556.778670:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778675:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778679:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778683:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.778689:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696556.778702:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145661952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.778714:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778722:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696556.778725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778731:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778736:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778740:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.778746:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696556.778764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145662080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.778799:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778804:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696556.778806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778828:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778835:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778846:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.778854:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696556.778868:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145662208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.778885:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778890:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696556.778893:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778898:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778903:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778908:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.778917:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696556.778936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145662336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.778958:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.778964:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696556.778968:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.778977:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.778984:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.778992:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.778999:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696556.779018:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145662464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.779030:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.779034:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696556.779036:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.779040:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.779047:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.779055:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.779062:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696556.779077:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145662592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696556.779102:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696556.779108:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696556.779113:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696556.779123:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4456 00000100:00080000:2.0:1787696556.779130:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696556.779138:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696556.779145:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696556.779164:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145662720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696557.674535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696560.746545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696561.898438:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898446:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696561.898450:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898457:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898467:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898472:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696561.898533:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145662848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898557:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696561.898559:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898562:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898565:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898567:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898575:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696561.898585:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145662976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696561.898596:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898599:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898601:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898606:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898613:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696561.898624:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145663104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898631:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898634:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696561.898636:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898638:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898640:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898642:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898646:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696561.898652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145663232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696561.898662:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898669:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898673:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898676:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696561.898687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145663360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898696:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696561.898698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898701:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898704:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898710:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696561.898729:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145663488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898745:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898751:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696561.898756:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898764:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898771:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898778:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898783:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696561.898797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145663616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898815:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696561.898821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898826:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898832:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898837:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696561.898854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145663744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696561.898861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.898863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696561.898864:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.898868:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4461 00000100:00080000:3.0:1787696561.898873:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696561.898876:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696561.898880:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696561.898890:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145663872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696564.842586:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696568.938546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696573.034591:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696577.130540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696581.226545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696585.322527:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696589.418532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696593.514492:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696597.610539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696597.738463:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738476:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696597.738480:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738493:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738505:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738514:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696597.738556:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145670912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738581:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696597.738585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738589:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696597.738623:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145671040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696597.738657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738662:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738666:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738669:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696597.738690:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145671168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738697:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738699:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696597.738702:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738705:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738710:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738715:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696597.738726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145671296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696597.738736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738739:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738741:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738744:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738746:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696597.738751:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145671424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696597.738764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738770:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738775:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738783:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696597.738823:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145671552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738840:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738846:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696597.738850:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738858:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738880:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738886:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696597.738909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145671680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738926:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696597.738928:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.738933:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.738936:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.738941:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.738946:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696597.738958:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145671808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696597.738982:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696597.738989:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696597.738993:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696597.739002:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4497 00000100:00080000:3.0:1787696597.739008:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696597.739015:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696597.739020:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696597.739035:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145671936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696601.706532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696602.858453:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858462:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696602.858466:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858475:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858484:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858490:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696602.858524:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145672064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858546:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858549:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696602.858550:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858554:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858556:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858558:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858566:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696602.858575:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145672192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858588:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696602.858591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858594:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858598:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858602:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696602.858612:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145672320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858623:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858627:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696602.858629:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858632:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858637:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858641:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696602.858661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145672448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858673:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858679:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696602.858681:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858685:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858690:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858693:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696602.858707:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145672576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696602.858717:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858720:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858722:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858725:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858728:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696602.858739:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145672704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858785:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696602.858789:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858792:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858796:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858801:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696602.858829:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145672832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858862:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858865:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696602.858866:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858871:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858875:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858880:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858886:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696602.858907:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145672960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696602.858933:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.858937:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696602.858939:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.858945:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4502 00000100:00080000:3.0:1787696602.858952:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696602.858959:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696602.858966:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696602.858985:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145673088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696605.802560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696607.978493:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978501:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696607.978505:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978512:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978522:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978528:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696607.978560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145673216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978584:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696607.978588:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978596:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978603:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696607.978611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145673344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978619:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978622:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696607.978623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978626:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978627:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978630:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696607.978648:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145673472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978656:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978659:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696607.978660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978663:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978665:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978667:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696607.978678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145673600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978701:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696607.978702:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978705:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978710:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978713:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696607.978722:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145673728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978730:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978732:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696607.978733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978737:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978739:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978741:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978745:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696607.978752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145673856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978762:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978768:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696607.978772:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978782:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978788:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978793:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978800:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696607.978821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145673984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978841:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696607.978847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978853:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978859:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978863:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978868:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696607.978884:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145674112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696607.978904:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696607.978909:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696607.978914:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696607.978922:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4507 00000100:00080000:3.0:1787696607.978924:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696607.978927:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696607.978930:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696607.978936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145674240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696609.898596:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696613.994559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696618.090559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696618.218446:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696618.218457:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218467:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218476:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218482:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696618.218532:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145675520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218553:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218555:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696618.218557:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218560:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218562:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218565:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218572:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696618.218583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145675648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218590:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696618.218594:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218599:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218604:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696618.218615:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145675776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218629:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218631:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696618.218632:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218635:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218637:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218639:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696618.218651:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145675904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696618.218661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218663:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218667:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696618.218677:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145676032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218683:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218687:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696618.218689:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218693:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218697:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218702:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696618.218723:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145676160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218737:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696618.218743:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218748:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218751:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218754:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696618.218776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145676288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218792:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696618.218794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218797:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218800:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218803:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696618.218817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145676416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696618.218831:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696618.218834:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696618.218835:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696618.218838:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4517 00000100:00080000:2.0:1787696618.218840:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696618.218843:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696618.218846:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696618.218856:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145676544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000400:00000100:3.0:1787696621.290664:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696621.290709:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696621.290735:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290783:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290798:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290809:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290820:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290829:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290840:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290853:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696621.290864:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00010000:00010000:3.0:1787696622.186549:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696623.338483:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338495:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696623.338500:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338511:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338521:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338530:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696623.338567:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145676672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.338597:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338601:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696623.338603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338608:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338612:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338617:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.338627:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696623.338642:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145676800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.338654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696623.338659:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338663:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338667:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338671:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.338676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696623.338698:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145676928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.338712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696623.338718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338723:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338728:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338733:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.338740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696623.338757:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145677056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.338779:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338783:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696623.338785:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338791:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338796:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338816:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.338825:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696623.338844:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145677184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.338855:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338860:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696623.338862:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338867:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338873:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338880:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.338887:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696623.338908:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145677312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.338935:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.338940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696623.338943:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.338951:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.338965:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.338977:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.338986:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696623.339011:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145677440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.339045:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.339048:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696623.339050:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.339055:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.339061:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.339072:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.339088:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696623.339112:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145677568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696623.339163:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696623.339179:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696623.339183:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696623.339193:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4522 00000100:00080000:2.0:1787696623.339205:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696623.339222:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696623.339243:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696623.339269:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145677696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787696626.282557:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696628.459151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145678208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696628.459161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696628.459163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696628.459165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145678208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696628.459174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145678336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696628.459187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696628.459194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696628/real 1787696628] req@ffff8f51ace92d80 x1874531145678336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696628.459202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145678336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696628.459209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696628.459213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696628.459216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145678336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696628.459222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696628.459243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696628.459255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696628/real 1787696628] req@ffff8f51ace90000 x1874531145678464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696628.459267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145678464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696628.459273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696628.459275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696628.459278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696628.459282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696628.459296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696628.459303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696628/real 1787696628] req@ffff8f51ace93480 x1874531145678592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696628.459311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145678592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696628.459319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696628.459322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696628.459326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696628.459344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696628.459399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696628.459412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696628/real 1787696628] req@ffff8f51ace92680 x1874531145678720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696628.459425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145678720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696628.459444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696628.459448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696628.459452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696628.459469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696628.459485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696628.459493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696628/real 1787696628] req@ffff8f51ace90e00 x1874531145678848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696628.459501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145678848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696628.459507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696628.459509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696628.459511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145678848:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696629.354644:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696630.378513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696632.426622:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696633.450534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696633.578610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145678976:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696633.578678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.578725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace91180 x1874531145678976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.578762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145678976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.578788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.578798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.578808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145678976:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696633.578823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696633.578860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.578877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace90380 x1874531145679104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.578898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145679104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.578908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.578913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.578917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696633.578944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696633.578967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.578979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace93800 x1874531145679232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.578993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145679232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696633.579015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696633.579036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.579047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace90700 x1874531145679360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.579058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145679360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696633.579086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696633.579104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.579115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace93b80 x1874531145679488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.579127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145679488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145679488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696633.579149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145679616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696633.579168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.579179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace93100 x1874531145679616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.579191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145679616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145679616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696633.579213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145679744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696633.579231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.579241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace90e00 x1874531145679744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.579275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145679744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145679744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696633.579298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145679872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696633.579317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.579327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace92680 x1874531145679872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.579339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145679872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145679872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696633.579389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145680000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696633.579407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696633.579419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696633/real 1787696633] req@ffff8f51ace93480 x1874531145680000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696633.579431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145680000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696633.579440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696633.579444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696633.579447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145680000:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696634.474484:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696636.522485:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696637.546534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696638.570478:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696638.698498:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.698506:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696638.698510:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696638.698521:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698533:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.698541:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696638.698578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145680128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696638.698609:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.698614:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696638.698616:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696638.698621:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698626:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.698631:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696638.698641:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696638.698654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145680128:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696638.698659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145680256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696638.698684:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.698689:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696638.698691:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696638.698696:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698703:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.698707:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787696638.698712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787696638.698716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696638.698733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145680384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696638.698744:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.698748:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696638.698750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696638.698755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183646d80 x1874531145680128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696638.698757:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698764:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.698771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696638.698776:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696638.698778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145680128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.698788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787696638.698789:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145680512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696638.698794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.698802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145680128:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696638.698806:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.698811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696638.698821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696638.698823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680256:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696638.698833:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698840:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787696638.698845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696638.698847:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696638.698855:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696638.698859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183645180 x1874531145680256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696638.698876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145680256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696638.698877:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145680640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696638.698897:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696638.698902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696638.698903:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696638.698907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680256:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696638.698908:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696638.698913:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696638.698917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680384:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696638.698920:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698926:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.698930:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696638.698934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696638.698944:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145680768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696638.698952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696638.698957:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.698961:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696638.698962:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696638.698966:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.698969:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696638.698973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183646680 x1874531145680384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696638.698974:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696638.698981:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696638.698994:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145680896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696638.698996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145680384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.699020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696638.699023:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696638.699025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696638.699028:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787696638.699031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680384:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696638.699032:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696638.699040:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00100000:2.0:1787696638.699044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680512:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787696638.699045:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.699052:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696638.699056:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696638.699067:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145681024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696638.699074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696638.699081:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696638.699084:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696638.699085:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696638.699088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183647480 x1874531145680512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696638.699091:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4538 00000100:00080000:1.0:1787696638.699097:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696638.699103:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696638.699106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145680512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787696638.699108:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787696638.699119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787696638.699121:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145681152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696638.699127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.699136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696638.699175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696638.699198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696638.699208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183646d80 x1874531145680640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696638.699216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145680640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.699222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696638.699224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.699227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145680640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696638.699230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145680768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696638.699243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696638.699249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183647100 x1874531145680768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696638.699256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145680768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.699262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696638.699264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.699266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145680768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696638.699269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145680896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696638.699278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696638.699284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183647800 x1874531145680896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696638.699292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145680896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.699304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696638.699306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.699311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145680896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696638.699314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145681024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696638.699326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696638.699333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183644000 x1874531145681024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696638.699341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145681024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.699346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696638.699347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.699349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145681024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696638.699374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145681152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696638.699388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696638.699395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696638/real 1787696638] req@ffff8f5183644700 x1874531145681152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696638.699402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145681152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696638.699408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696638.699410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696638.699412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145681152:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696640.618529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696641.642528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696642.666565:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696643.818462:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818470:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696643.818473:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818482:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696643.818501:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696643.818548:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145681280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818577:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696643.818584:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818589:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696643.818599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696643.818608:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696643.818620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145681280:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696643.818622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145681408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818636:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696643.818640:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818643:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818646:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696643.818648:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696643.818651:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696643.818661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145681536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818673:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696643.818677:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818679:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818682:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787696643.818684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696643.818687:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696643.818693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696643.818702:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145681664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818708:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818710:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696643.818712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818715:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818717:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696643.818720:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696643.818724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696643.818728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183644380 x1874531145681280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696643.818733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145681792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818755:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696643.818758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696643.818759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145681280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696643.818765:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818771:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696643.818778:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787696643.818780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787696643.818781:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787696643.818788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.818793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145681280:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696643.818794:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145681920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818804:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696643.818807:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818809:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818812:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696643.818816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681408:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696643.818817:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696643.818824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696643.818835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145682048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696643.818844:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787696643.818846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696643.818848:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696643.818851:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696643.818855:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818859:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696643.818863:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696643.818866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696643.818870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183645c00 x1874531145681408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696643.818874:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145682176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696643.818902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145681408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696643.818905:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696643.818913:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696643.818916:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696643.818920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696643.818921:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4543 00000100:00080000:1.0:1787696643.818926:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696643.818927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696643.818932:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696643.818935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681408:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787696643.818937:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696643.818944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696643.818952:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145682304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696643.818962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.818973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183646a00 x1874531145681536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.818982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145681536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.818988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.818990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.818993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696643.818997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696643.819008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.819014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183644e00 x1874531145681664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.819022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145681664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.819027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.819029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.819031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696643.819034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696643.819043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.819049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183647b80 x1874531145681792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.819056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145681792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.819061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.819063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.819064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145681792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696643.819091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145681920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696643.819107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.819114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183644700 x1874531145681920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.819122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145681920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.819128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.819130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.819132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145681920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696643.819146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145682048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696643.819156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.819163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183644000 x1874531145682048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.819170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145682048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.819175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.819177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.819179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145682048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696643.819182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145682176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696643.819192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.819197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183647800 x1874531145682176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.819204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145682176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.819214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.819216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.819217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145682176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696643.819221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145682304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696643.819230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696643.819235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696643/real 1787696643] req@ffff8f5183647100 x1874531145682304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696643.819242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145682304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696643.819247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696643.819249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696643.819251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145682304:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696644.714563:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696645.738537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696646.762502:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696648.810577:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696648.938513:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938522:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696648.938525:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938533:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938542:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938547:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696648.938582:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145682432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938607:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696648.938612:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938615:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938618:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938620:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.938628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696648.938637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145682560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696648.938652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938660:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938670:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696648.938673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145682432:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787696648.938678:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696648.938697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145682688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938707:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938709:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696648.938712:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938714:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938718:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938721:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.938725:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696648.938738:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145682816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938744:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938747:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696648.938748:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938750:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938753:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938755:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.938757:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696648.938767:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145682944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938775:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938780:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696648.938781:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696648.938783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696648.938793:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938800:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938805:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.938811:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696648.938825:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145683072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938835:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938838:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696648.938839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938845:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938847:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.938851:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696648.938861:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145683200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938868:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696648.938874:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696648.938875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704c000 x1874531145682432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696648.938876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696648.938888:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.938894:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696648.938903:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.938914:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696648.938930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145682432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696648.938935:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145683328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696648.938987:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696648.938991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696648.938991:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696648.938997:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696648.939008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696648.939009:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4548 00000100:00080000:2.0:1787696648.939030:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696648.939034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145682432:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696648.939041:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696648.939053:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696648.939061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696648.939069:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145683456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696648.939099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704d500 x1874531145682560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145682560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696648.939187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696648.939206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704f480 x1874531145682688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145682688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696648.939260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696648.939278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704e300 x1874531145682816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145682816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696648.939331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696648.939347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704d180 x1874531145682944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145682944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145682944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696648.939417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696648.939435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704c700 x1874531145683072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145683072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696648.939488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696648.939504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704dc00 x1874531145683200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145683200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696648.939546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696648.939563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704fb80 x1874531145683328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145683328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696648.939607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696648.939623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696648.939632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696648/real 1787696648] req@ffff8f518704ed80 x1874531145683456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696648.939644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145683456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696648.939653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696648.939656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696648.939659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145683456:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696649.834542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696650.858553:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696652.906568:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696653.930505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696654.058654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145683584:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696654.058713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.058759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace90000 x1874531145683584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.058784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145683584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.058794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.058798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.058802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145683584:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696654.058830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145683712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696654.058852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.058864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace92d80 x1874531145683712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.058879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145683712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.058908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.058915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.058920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145683712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696654.058928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145683840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696654.058956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.058972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace92300 x1874531145683840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.058999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145683840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145683840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696654.059036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145683968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696654.059060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.059073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace91500 x1874531145683968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.059100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145683968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145683968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696654.059144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145684096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696654.059162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.059174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace91c00 x1874531145684096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.059204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145684096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145684096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696654.059244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696654.059263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.059273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace91880 x1874531145684224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.059286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145684224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696654.059341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696654.059376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.059385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace90000 x1874531145684352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.059393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145684352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696654.059412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696654.059424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.059431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace93480 x1874531145684480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.059438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145684480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696654.059450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696654.059460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696654.059465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696654/real 1787696654] req@ffff8f51ace92680 x1874531145684608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696654.059472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145684608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696654.059486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696654.059488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696654.059490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145684608:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696654.954554:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696657.002515:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696658.026486:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696659.050555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696659.178449:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178461:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696659.178465:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178474:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178484:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178491:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696659.178521:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145684736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696659.178543:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178546:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696659.178548:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178551:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178554:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178557:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178565:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696659.178577:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145684864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696659.178579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145684736:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696659.178589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178592:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696659.178594:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178597:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178601:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696659.178611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145684992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696659.178616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696659.178620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178623:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178625:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178627:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178631:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696659.178637:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145685120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696659.178644:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178646:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696659.178647:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178650:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178653:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787696659.178657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696659.178658:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696659.178672:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145685248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696659.178679:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178681:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696659.178683:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178685:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178688:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178690:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696659.178699:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145685376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696659.178706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178708:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696659.178710:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696659.178713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183646d80 x1874531145684736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696659.178714:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178719:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178733:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696659.178742:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145685504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696659.178747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145684736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696659.178751:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696659.178761:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178768:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00080000:1.0:1787696659.178772:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696659.178773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696659.178775:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787696659.178782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.178790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145684736:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696659.178797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145685632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696659.178823:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696659.178827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696659.178829:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696659.178835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4558 00000100:00100000:2.0:1787696659.178837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145684864:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696659.178840:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696659.178848:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696659.178854:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696659.178868:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145685760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696659.178878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.178897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183647480 x1874531145684864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.178921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145684864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.178930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.178934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.178937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145684864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696659.178944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145684992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696659.178966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.178978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183646680 x1874531145684992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.178989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145684992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.178996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.178999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145684992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696659.179007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145685120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696659.179024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.179034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183645180 x1874531145685120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.179044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145685120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.179060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.179063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145685120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696659.179072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145685248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696659.179090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.179100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183647100 x1874531145685248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.179119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145685248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.179128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.179132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145685248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696659.179141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696659.179162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.179175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183647800 x1874531145685376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.179197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145685376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.179204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.179206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696659.179213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696659.179228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.179236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183644000 x1874531145685504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.179243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145685504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.179252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.179254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696659.179260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696659.179269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.179275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183644700 x1874531145685632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.179282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145685632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.179287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.179288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696659.179305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696659.179322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696659.179332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696659/real 1787696659] req@ffff8f5183647b80 x1874531145685760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696659.179345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145685760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696659.179376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696659.179379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696659.179382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145685760:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696661.098569:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696662.122513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696663.146498:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696664.298425:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298446:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696664.298451:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298461:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298472:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298480:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696664.298513:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145685888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696664.298535:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298538:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696664.298540:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298543:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298546:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298549:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298559:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696664.298572:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145686016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696664.298584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145685888:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696664.298597:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298602:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696664.298605:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298610:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298614:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298618:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696664.298633:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145686144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696664.298641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298643:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696664.298644:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787696664.298649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696664.298653:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298660:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696664.298672:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145686272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696664.298679:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298681:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696664.298683:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298686:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298688:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696664.298691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183644e00 x1874531145685888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696664.298694:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298700:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696664.298726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145686400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696664.298728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145685888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.298755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696664.298759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696664.298761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696664.298764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696664.298768:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696664.298768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145685888:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696664.298776:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298779:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298783:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298787:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696664.298799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696664.298800:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145686528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696664.298813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298816:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696664.298817:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298820:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298824:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298826:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298830:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787696664.298831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696664.298842:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145686656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696664.298848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183646a00 x1874531145686016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696664.298856:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696664.298866:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298873:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00100000:2.0:1787696664.298877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145686016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696664.298878:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696664.298885:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298891:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787696664.298894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.298901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696664.298902:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145686784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696664.298911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686016:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696664.298918:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696664.298923:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696664.298925:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696664.298933:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4563 00000100:00080000:1.0:1787696664.298937:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696664.298940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686144:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696664.298945:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696664.298966:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787696664.298972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696664.298981:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145686912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696664.298990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183645c00 x1874531145686144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145686144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696664.299017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696664.299032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696664.299039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183644380 x1874531145686272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145686272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696664.299060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696664.299069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696664.299075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183647b80 x1874531145686400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145686400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145686400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696664.299112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696664.299133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696664.299145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183644700 x1874531145686528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145686528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696664.299169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696664.299183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696664.299203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183644000 x1874531145686656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145686656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696664.299237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696664.299263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696664.299272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183647800 x1874531145686784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145686784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696664.299292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696664.299309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696664.299315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696664/real 1787696664] req@ffff8f5183647100 x1874531145686912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696664.299323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145686912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696664.299328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696664.299330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696664.299332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145686912:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696665.194554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696666.218589:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696667.242508:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696669.290490:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696669.418465:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418475:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696669.418480:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696669.418490:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:2.0:1787696669.418507:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418515:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696669.418549:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145687040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696669.418581:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418585:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696669.418587:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696669.418592:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:2.0:1787696669.418596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418600:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418610:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696669.418640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145687168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696669.418651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145687040:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696669.418653:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696669.418664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696669.418668:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:2.0:1787696669.418672:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418676:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696669.418694:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145687296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696669.418711:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418714:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696669.418716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696669.418722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696669.418731:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:2.0:1787696669.418737:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418742:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418747:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696669.418759:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145687424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696669.418770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704f800 x1874531145687040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696669.418770:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696669.418780:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696669.418789:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:2.0:1787696669.418794:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418798:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418803:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696669.418804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145687040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696669.418817:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145687552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696669.418836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696669.418838:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696669.418843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696669.418845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696669.418849:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696669.418852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145687040:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696669.418856:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:2.0:1787696669.418861:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418865:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418870:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696669.418880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696669.418882:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145687680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696669.418896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418900:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696669.418902:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696669.418906:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4569 00000800:00000100:1.0:1787696669.418911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696669.418912:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696669.418919:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418924:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696669.418929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704ea00 x1874531145687168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696669.418937:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145687808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696669.418950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145687168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696669.418952:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.418960:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696669.418964:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696669.418969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696669.418972:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00080000:1.0:1787696669.418977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696669.418978:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696669.418983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687168:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696669.418985:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696669.418993:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696669.418998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696669.419016:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145687936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696669.419023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704d880 x1874531145687296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696669.419040:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696669.419049:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696669.419053:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696669.419060:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4569 00000100:00100000:1.0:1787696669.419063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145687296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696669.419070:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696669.419074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696669.419078:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696669.419082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787696669.419086:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696669.419088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696669.419098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696669.419102:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145688064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696669.419122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704df80 x1874531145687424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696669.419148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145687424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696669.419156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696669.419160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696669.419163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696669.419169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696669.419184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704ce00 x1874531145687552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696669.419205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145687552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696669.419213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696669.419216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696669.419219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145687552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696669.419235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145687680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696669.419251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704e680 x1874531145687680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696669.419274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145687680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696669.419283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696669.419287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696669.419290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145687680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696669.419296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145687808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696669.419311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704c380 x1874531145687808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696669.419332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145687808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696669.419340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696669.419344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696669.419347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145687808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696669.419372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145687936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696669.419390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704f800 x1874531145687936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696669.419413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145687936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696669.419428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696669.419431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696669.419435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145687936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696669.419441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145688064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696669.419455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696669.419464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696669/real 1787696669] req@ffff8f518704ed80 x1874531145688064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696669.419477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145688064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696669.419495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696669.419499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696669.419502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145688064:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696670.314490:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696671.338542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696673.386535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696674.410618:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696674.538739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145688192:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696674.538816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.538880:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace90e00 x1874531145688192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.538929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145688192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.538978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.538988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.538995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145688192:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696674.539032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696674.539070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace93100 x1874531145688320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145688320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696674.539162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696674.539192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace93b80 x1874531145688448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145688448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696674.539270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696674.539302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace90700 x1874531145688576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145688576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696674.539348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696674.539397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace93800 x1874531145688704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145688704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145688704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696674.539435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145688832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696674.539447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace90380 x1874531145688832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145688832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145688832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696674.539476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145688960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696674.539486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace91180 x1874531145688960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145688960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145688960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696674.539511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145689088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696674.539521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace90e00 x1874531145689088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145689088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145689088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696674.539582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145689216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696674.539598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696674.539605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696674/real 1787696674] req@ffff8f51ace92680 x1874531145689216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696674.539613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145689216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696674.539618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696674.539621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696674.539623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145689216:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696675.434518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696677.482515:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696678.506526:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696679.530542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696679.658671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145689344:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696679.658742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.658790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace93480 x1874531145689344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.658822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145689344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.658838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.658849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.658856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145689344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696679.658883:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696679.658915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.658936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace90000 x1874531145689472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.658963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145689472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.658978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.658985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.658993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696679.659006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696679.659034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace91880 x1874531145689600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145689600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696679.659119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696679.659147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace91c00 x1874531145689728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145689728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696679.659202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696679.659219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace91500 x1874531145689856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145689856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145689856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696679.659273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145689984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696679.659290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace92300 x1874531145689984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145689984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145689984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696679.659342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145690112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696679.659389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace92d80 x1874531145690112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145690112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145690112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696679.659435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145690240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696679.659472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace93480 x1874531145690240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145690240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145690240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696679.659527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145690368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696679.659546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696679.659557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696679/real 1787696679] req@ffff8f51ace92680 x1874531145690368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696679.659570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145690368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696679.659578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696679.659581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696679.659584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145690368:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696681.578627:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696682.602524:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696683.626649:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696684.778519:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778529:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696684.778534:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778546:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778558:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778566:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696684.778614:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145690496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696684.778646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778650:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778652:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778655:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696684.778677:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145690624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778685:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778688:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696684.778689:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778692:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778694:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778696:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778699:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696684.778710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145690496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696684.778719:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145690752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778737:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696684.778740:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778744:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778748:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778751:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778757:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696684.778767:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145690880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778774:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778785:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696684.778788:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778791:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778794:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778796:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778800:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696684.778808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145691008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778816:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696684.778821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778826:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778828:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778832:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778836:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696684.778847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145691136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696684.778853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145690624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696684.778858:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778862:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696684.778865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778874:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778877:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778881:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778884:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696684.778898:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145691264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778907:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696684.778911:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778914:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778917:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778922:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696684.778923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145690752:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787696684.778925:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696684.778937:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145691392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696684.778946:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696684.778948:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696684.778950:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696684.778953:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4584 00000100:00080000:2.0:1787696684.778956:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696684.778959:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696684.778962:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696684.778970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145690880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696684.778974:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145691520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696684.779019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145691008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696684.779044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696684.779075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696684.779102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696684.779121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691520:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696685.674621:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696686.698534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696687.722620:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696687.850680:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696687.850741:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696687.850761:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696687.850823:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696687.850841:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696687.850854:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696687.850859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704ed80 x1874531145691520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696687.850863:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696687.850872:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696687.850890:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696687.850891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145691520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696687.850900:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787696687.850906:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:3.0:1787696687.850908:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787696687.850911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.850916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696687.850925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704c700 x1874531145690752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.850940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145690752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.850949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.850953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.850957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145690752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696687.850968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704d180 x1874531145690880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.850981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145690880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.850989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.850993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.850997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145690880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696687.851005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704e300 x1874531145691008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.851019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145691008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.851029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.851032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.851036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145691008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696687.851045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704f480 x1874531145691136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.851058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145691136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.851069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.851073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.851077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696687.851085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704d500 x1874531145691264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.851097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145691264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.851104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.851108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.851111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696687.851119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704fb80 x1874531145690496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.851133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145690496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.851143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.851146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.851150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145690496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696687.851157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704dc00 x1874531145690624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.851169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145690624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.851177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.851181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.851184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145690624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696687.851224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696684/real 1787696687] req@ffff8f518704c000 x1874531145691392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696687.851237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145691392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696687.851246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696687.851250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696687.851253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145691392:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696689.770553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696689.898437:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898445:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696689.898448:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898457:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898465:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.898470:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696689.898498:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145691648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696689.898523:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898527:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696689.898529:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898534:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898538:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.898542:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696689.898551:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696689.898568:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145691776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696689.898573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145691648:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696689.898585:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898589:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696689.898590:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898594:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.898600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696689.898606:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696689.898620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145691904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696689.898636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696689.898642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898647:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4589 00000800:00000100:1.0:1787696689.898649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696689.898654:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.898661:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696689.898666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696689.898681:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145692032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696689.898691:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898694:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696689.898696:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898704:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.898708:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696689.898713:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696689.898722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704f800 x1874531145691648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696689.898729:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145692160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696689.898745:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898757:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696689.898759:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696689.898764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145691648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696689.898767:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898774:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.898783:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696689.898788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787696689.898789:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696689.898797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.898805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145691648:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696689.898808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145692288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696689.898820:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898823:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696689.898825:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898829:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898833:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696689.898838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145691776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696689.898841:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696689.898850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696689.898863:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145692416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696689.898873:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696689.898878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696689.898881:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696689.898884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898890:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.898895:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696689.898900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704c380 x1874531145691776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696689.898903:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696689.898912:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696689.898931:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145692544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696689.898934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145691776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.898959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.898964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.898973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145691776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696689.898978:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696689.898983:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696689.898984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145691904:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696689.898986:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696689.898995:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4589 00000100:00080000:2.0:1787696689.899001:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696689.899006:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696689.899017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696689.899019:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696689.899036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704e680 x1874531145691904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696689.899037:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145692672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696689.899083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145691904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145691904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696689.899109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145692032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696689.899133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696689.899146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704ce00 x1874531145692032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696689.899158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145692032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145692032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696689.899185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145692160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696689.899206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696689.899218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704df80 x1874531145692160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696689.899230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145692160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145692160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696689.899250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696689.899283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696689.899298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704d880 x1874531145692288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696689.899311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145692288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696689.899380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696689.899404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696689.899418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704ea00 x1874531145692416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696689.899431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145692416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696689.899452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696689.899470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696689.899482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704f800 x1874531145692544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696689.899493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145692544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696689.899514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696689.899533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696689.899545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696689/real 1787696689] req@ffff8f518704c000 x1874531145692672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696689.899558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145692672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696689.899566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696689.899569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696689.899573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145692672:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696690.794514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696691.818522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696693.866565:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696694.890504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696695.018483:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696695.018493:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696695.018497:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.018508:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.018520:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696695.018528:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696695.018566:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145692800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696695.018597:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696695.018601:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696695.018604:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.018609:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.018613:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696695.018617:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696695.018630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696695.018630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145692800:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696695.018649:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145692928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696695.018663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696695.018667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696695.018670:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.018673:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.018680:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696695.018685:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787696695.018689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787696695.018707:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696695.018722:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145693056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696695.018734:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696695.018735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183645180 x1874531145692800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696695.018741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696695.018747:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.018755:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.018760:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696695.018760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145692800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696695.018770:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787696695.018775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787696695.018778:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787696695.018781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.018786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145692800:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696695.018798:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145693184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696695.018808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145692928:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696695.018813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696695.018818:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696695.018822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.018828:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4594 00000800:00000100:2.0:1787696695.018829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696695.018837:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696695.018844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183646680 x1874531145692928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696695.018845:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696695.018855:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696695.018860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145692928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696695.018868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787696695.018873:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145693312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696695.018873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.018879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145692928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696695.018888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145693056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696695.018896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696695.018900:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696695.018902:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787696695.018907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696695.018911:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.018917:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696695.018919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183647480 x1874531145693056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696695.018924:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696695.018934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696695.018943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145693056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696695.018951:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145693440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696695.018960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.018969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696695.018970:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696695.018976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145693056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696695.018977:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696695.018982:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.018988:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.018992:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696695.018994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145693184:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787696695.019000:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696695.019007:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787696695.019019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696695.019024:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145693568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696695.019036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183646d80 x1874531145693184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696695.019038:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696695.019046:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696695.019051:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.019059:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.019063:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696695.019067:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696695.019073:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696695.019074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145693184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696695.019088:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145693696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696695.019089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.019096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.019103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145693184:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787696695.019114:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696695.019114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145693312:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787696695.019121:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696695.019124:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696695.019130:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4594 00000100:00080000:1.0:1787696695.019135:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787696695.019136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696695.019143:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696695.019148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183645180 x1874531145693312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787696695.019150:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696695.019163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145693312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696695.019170:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145693824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696695.019172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.019179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.019204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145693312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696695.019211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696695.019234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696695.019245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183647100 x1874531145693440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696695.019253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145693440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696695.019259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.019261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.019263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696695.019273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696695.019285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696695.019291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183647800 x1874531145693568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696695.019298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145693568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696695.019304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.019305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.019307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696695.019311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696695.019320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696695.019325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183644000 x1874531145693696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696695.019332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145693696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696695.019337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.019339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.019340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696695.019344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696695.019370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696695.019378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696695/real 1787696695] req@ffff8f5183644700 x1874531145693824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696695.019386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145693824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696695.019392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696695.019394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696695.019396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145693824:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696695.914536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696697.962561:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696698.986537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696700.010499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696700.138419:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138428:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696700.138433:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696700.138441:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:1.0:1787696700.138450:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138457:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696700.138488:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145693952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696700.138510:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138513:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696700.138514:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696700.138518:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:1.0:1787696700.138520:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138522:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138530:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696700.138538:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145694080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696700.138548:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138550:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696700.138552:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696700.138553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145693952:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696700.138566:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:1.0:1787696700.138575:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138580:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696700.138601:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145694208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696700.138618:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787696700.138624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696700.138625:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696700.138629:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696700.138635:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:1.0:1787696700.138640:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138645:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138650:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696700.138663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145694336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696700.138670:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696700.138674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696700.138679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183647b80 x1874531145693952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696700.138680:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:1.0:1787696700.138686:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138693:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696700.138707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145693952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696700.138708:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145694464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696700.138729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138735:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696700.138736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696700.138738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696700.138740:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:2.0:1787696700.138745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696700.138746:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696700.138753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145693952:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696700.138754:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138759:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696700.138769:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145694592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696700.138780:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696700.138785:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696700.138789:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00100000:2.0:1787696700.138794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696700.138795:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138804:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138810:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696700.138825:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145694720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696700.138831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696700.138840:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696700.138843:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696700.138844:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696700.138850:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00100000:2.0:1787696700.138851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183644380 x1874531145694080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696700.138853:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138861:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696700.138870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145694080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696700.138878:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145694848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696700.138898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.138905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696700.138911:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696700.138913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696700.138915:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696700.138919:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787696700.138924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694208:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696700.138925:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4599 00000100:00080000:1.0:1787696700.138930:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696700.138933:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696700.138937:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696700.138946:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145694976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696700.138960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.138976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183645c00 x1874531145694208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.138991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145694208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696700.139011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696700.139027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.139035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183646a00 x1874531145694336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.139043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145694336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696700.139056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696700.139066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.139072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183644e00 x1874531145694464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.139080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145694464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145694464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696700.139091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696700.139118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.139126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183644700 x1874531145694592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.139134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145694592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696700.139158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696700.139169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.139176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183644000 x1874531145694720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.139184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145694720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696700.139198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696700.139215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.139227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183647800 x1874531145694848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.139239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145694848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696700.139257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696700.139272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696700.139279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696700/real 1787696700] req@ffff8f5183647100 x1874531145694976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696700.139287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145694976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696700.139292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696700.139294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696700.139296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145694976:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696702.058526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696703.082560:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696704.106499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787696705.258599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145695104:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696705.258676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.258712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace90e00 x1874531145695104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.258735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145695104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.258759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.258764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.258770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145695104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696705.258802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696705.258837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.258869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace91180 x1874531145695232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.258899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145695232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.258934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.258941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.258951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696705.258966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696705.259023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace90380 x1874531145695360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145695360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696705.259126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696705.259158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace93800 x1874531145695488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145695488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696705.259199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696705.259216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace90700 x1874531145695616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145695616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145695616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696705.259262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145695744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696705.259286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace93b80 x1874531145695744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145695744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145695744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696705.259320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145695872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696705.259331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace93100 x1874531145695872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145695872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145695872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696705.259391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145696000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696705.259410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace90e00 x1874531145696000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145696000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145696000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696705.259459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145696128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696705.259480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696705.259492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696705/real 1787696705] req@ffff8f51ace92680 x1874531145696128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696705.259505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145696128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696705.259513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696705.259516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696705.259520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145696128:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696706.154516:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696707.178563:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696708.202499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696710.250468:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787696710.378620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145696256:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787696710.378696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.378743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace93480 x1874531145696256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.378781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145696256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.378793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.378800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.378807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145696256:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696710.378818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696710.378846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.378857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace92d80 x1874531145696384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.378867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145696384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.378873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.378875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.378878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696710.378895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696710.378906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.378913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace92300 x1874531145696512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.378960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145696512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.378970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.378973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.378977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696710.378983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696710.379004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.379017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace91500 x1874531145696640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.379030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145696640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.379039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.379043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.379045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696710.379050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696710.379065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.379072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace91c00 x1874531145696768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.379080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145696768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.379086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.379088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.379090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145696768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696710.379093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145696896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696710.379103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.379109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace91880 x1874531145696896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.379116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145696896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.379122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.379123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.379125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145696896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696710.379129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145697024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696710.379138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.379144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace90000 x1874531145697024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.379151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145697024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.379156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.379158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.379160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145697024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696710.379164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145697152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696710.379181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.379192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace92680 x1874531145697152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.379204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145697152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.379212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.379216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.379219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145697152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696710.379223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145697280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696710.379238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696710.379246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696710/real 1787696710] req@ffff8f51ace90e00 x1874531145697280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696710.379253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145697280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696710.379259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696710.379261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696710.379263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145697280:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696711.274549:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696712.298530:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696714.346578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696715.370608:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696715.498480:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498492:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696715.498497:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498509:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498521:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498529:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696715.498572:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145697408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498603:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498606:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696715.498608:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498611:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498614:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498617:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696715.498635:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145697536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696715.498648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498653:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498656:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498658:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498661:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696715.498667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145697664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498686:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696715.498689:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696715.498690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145697408:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696715.498692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498698:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498704:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696715.498716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145697792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696715.498728:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498732:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498734:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498737:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696715.498750:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145697920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498756:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696715.498760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498764:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498768:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498771:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696715.498780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498782:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696715.498795:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145698048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696715.498808:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498810:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498814:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696715.498818:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696715.498827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704dc00 x1874531145697408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696715.498857:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145698176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696715.498865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145697408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696715.498877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498885:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696715.498888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696715.498889:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696715.498912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696715.498913:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.498932:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696715.498939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145697408:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696715.498941:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.498948:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696715.498960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696715.498962:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145698304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696715.498989:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696715.498992:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696715.498994:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696715.498998:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4615 00000100:00080000:2.0:1787696715.499002:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696715.499010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696715.499014:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696715.499020:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696715.499029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704fb80 x1874531145697536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696715.499035:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145698432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696715.499090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145697536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696715.499129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696715.499144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704d500 x1874531145697664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145697664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696715.499174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696715.499185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704f480 x1874531145697792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145697792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696715.499212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696715.499222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704e300 x1874531145697920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145697920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145697920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696715.499247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696715.499258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704d180 x1874531145698048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145698048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696715.499298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696715.499309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704c700 x1874531145698176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145698176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696715.499346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696715.499396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704ed80 x1874531145698304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145698304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696715.499432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696715.499444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696715.499450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696715/real 1787696715] req@ffff8f518704c000 x1874531145698432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696715.499457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145698432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696715.499462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696715.499464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696715.499466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145698432:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696716.394526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696718.442580:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696719.466510:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696720.490571:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787696720.618495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618504:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787696720.618508:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618516:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618525:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618531:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696720.618564:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145698560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696720.618596:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618599:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696720.618600:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618603:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618606:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618609:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618618:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696720.618627:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145698688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696720.618634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145698560:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696720.618637:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618643:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696720.618645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618648:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618650:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618656:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696720.618664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145698816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696720.618670:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696720.618674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618677:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618680:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618682:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618686:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696720.618697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145698944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787696720.618698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787696720.618707:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618710:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696720.618712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618715:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618719:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618721:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696720.618732:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145699072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696720.618753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696720.618753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183645180 x1874531145698560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696720.618763:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696720.618767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618776:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618780:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787696720.618784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145698560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696720.618786:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787696720.618798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.618803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696720.618810:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145699200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696720.618811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145698560:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787696720.618833:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787696720.618836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145698688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787696720.618843:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696720.618847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618853:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618859:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618863:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787696720.618867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618871:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696720.618880:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183646d80 x1874531145698688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696720.618886:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145699328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696720.618902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145698688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696720.618904:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618913:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696720.618916:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696720.618924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696720.618925:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618932:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696720.618933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787696720.618938:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696720.618941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145698688:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787696720.618943:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696720.618949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145698816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696720.618955:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145699456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696720.618962:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696720.618964:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696720.618966:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696720.618968:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4620 00000100:00080000:1.0:1787696720.618971:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696720.618974:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787696720.618982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787696720.618986:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696720.618995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183647480 x1874531145698816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696720.618999:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145699584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787696720.619030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145698816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145698816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696720.619056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145698944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696720.619074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696720.619082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183646680 x1874531145698944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696720.619090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145698944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145698944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696720.619108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145699072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696720.619118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696720.619124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183647100 x1874531145699072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696720.619131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145699072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145699072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696720.619144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696720.619154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696720.619159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183647800 x1874531145699200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696720.619166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145699200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696720.619192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787696720.619211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696720.619223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183644000 x1874531145699328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696720.619236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145699328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696720.619258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696720.619278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696720.619287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183644700 x1874531145699456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696720.619295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145699456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696720.619309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787696720.619326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696720.619337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696720/real 1787696720] req@ffff8f5183644e00 x1874531145699584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696720.619351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145699584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696720.619379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696720.619382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787696720.619384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145699584:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696722.538545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696723.562532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696724.586549:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696725.738474:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696725.738483:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696725.738486:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696725.738495:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738505:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738511:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696725.738542:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145699712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696725.738568:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696725.738572:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696725.738574:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696725.738579:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738583:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738587:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696725.738597:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696725.738603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145699712:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696725.738609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145699840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696725.738621:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696725.738624:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696725.738626:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696725.738629:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738633:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738636:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696725.738640:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696725.738648:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145699968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696725.738671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696725.738674:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787696725.738676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696725.738676:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696725.738680:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738687:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738690:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696725.738695:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696725.738705:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145700096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696725.738714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696725.738718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704f800 x1874531145699712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696725.738718:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696725.738720:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696725.738724:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738728:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738731:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696725.738735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696725.738737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145699712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696725.738745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145700224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696725.738748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.738752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696725.738754:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696725.738756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145699712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696725.738758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696725.738760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696725.738764:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738767:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738770:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696725.738773:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696725.738778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145699840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696725.738783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145700352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696725.738792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696725.738796:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696725.738797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696725.738800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696725.738801:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738804:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738808:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696725.738812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704ea00 x1874531145699840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696725.738812:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696725.738822:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145700480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696725.738825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145699840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696725.738839:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696725.738841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696725.738843:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696725.738844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696725.738845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696725.738848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145699840:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696725.738849:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00100000:1.0:1787696725.738854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145699968:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696725.738858:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696725.738862:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696725.738866:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696725.738870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696725.738876:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145700608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696725.738881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704d880 x1874531145699968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696725.738885:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696725.738889:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696725.738891:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696725.738892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145699968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696725.738894:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4625 00000100:00080000:2.0:1787696725.738898:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696725.738901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696725.738901:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696725.738904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787696725.738905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696725.738908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145699968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696725.738913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145700096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696725.738914:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145700736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696725.738931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696725.738942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704df80 x1874531145700096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696725.738955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145700096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696725.738963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.738966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696725.738970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145700096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696725.738975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145700224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696725.738990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696725.738997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704ce00 x1874531145700224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696725.739005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145700224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696725.739011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.739013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696725.739015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145700224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696725.739034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696725.739046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696725.739053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704e680 x1874531145700352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696725.739060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145700352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696725.739066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.739068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696725.739070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696725.739074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696725.739083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696725.739089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704c380 x1874531145700480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696725.739096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145700480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696725.739101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.739103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696725.739110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696725.739114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696725.739125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696725.739131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704f800 x1874531145700608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696725.739138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145700608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696725.739143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.739144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696725.739146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696725.739150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696725.739159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696725.739164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696725/real 1787696725] req@ffff8f518704c000 x1874531145700736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696725.739171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145700736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696725.739176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696725.739178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696725.739180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145700736:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696726.634558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696727.658502:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696728.682559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696730.730538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696730.858429:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858438:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696730.858442:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858452:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858464:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858471:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696730.858508:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145700864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696730.858549:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696730.858554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858558:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858562:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858578:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858587:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696730.858599:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145700992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696730.858609:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787696730.858615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145700864:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696730.858615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858622:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858627:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858630:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858633:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696730.858650:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145701120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696730.858659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858662:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696730.858664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858667:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858670:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858672:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858675:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696730.858683:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145701248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696730.858692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696730.858693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858698:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696730.858699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858703:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858706:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858709:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858712:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696730.858720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145701376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696730.858726:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696730.858731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696730.858733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704ed80 x1874531145700864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696730.858736:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858742:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858749:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696730.858758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145700864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696730.858766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145701504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696730.858769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696730.858777:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696730.858778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696730.858783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787696730.858785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145700864:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696730.858786:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858795:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858797:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858800:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858803:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696730.858809:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145701632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696730.858813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145700992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696730.858827:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858831:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696730.858833:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858839:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4630 00000800:00000100:1.0:1787696730.858841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696730.858847:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858854:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696730.858856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704c700 x1874531145700992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696730.858862:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696730.858869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145700992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696730.858879:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145701760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696730.858883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.858890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.858896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145700992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696730.858905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145701120:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696730.858905:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696730.858910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696730.858912:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696730.858924:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4630 00000100:00080000:2.0:1787696730.858932:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696730.858936:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696730.858940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696730.858944:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696730.858952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704d180 x1874531145701120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696730.858963:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145701888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696730.858964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145701120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.858989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.858994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.858998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145701120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696730.859003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145701248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696730.859016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696730.859023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704e300 x1874531145701248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696730.859030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145701248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.859035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.859037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.859039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145701248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696730.859053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145701376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696730.859072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696730.859084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704f480 x1874531145701376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696730.859098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145701376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.859107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.859110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.859113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145701376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696730.859120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696730.859141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696730.859154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704d500 x1874531145701504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696730.859167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145701504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.859176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.859180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.859184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696730.859204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696730.859222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696730.859231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704fb80 x1874531145701632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696730.859239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145701632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.859245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.859247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.859249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696730.859253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696730.859263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696730.859269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704dc00 x1874531145701760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696730.859276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145701760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.859281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.859283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.859284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696730.859288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696730.859301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696730.859311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696730/real 1787696730] req@ffff8f518704ed80 x1874531145701888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696730.859323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145701888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696730.859332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696730.859336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696730.859339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145701888:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696731.754509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696732.778484:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696734.826519:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696735.850517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696735.978663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145702016:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696735.978735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.978786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace93100 x1874531145702016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.978823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145702016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.978844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.978854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.978861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145702016:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696735.978876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696735.978908:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.978927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace93b80 x1874531145702144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.978948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145702144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.978965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.978970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.978975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696735.979000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696735.979020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace90700 x1874531145702272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145702272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696735.979066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696735.979085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace93800 x1874531145702400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145702400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696735.979131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696735.979150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace90380 x1874531145702528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145702528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145702528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696735.979196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145702656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696735.979215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace91180 x1874531145702656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145702656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145702656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696735.979261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145702784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696735.979279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace90e00 x1874531145702784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145702784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145702784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696735.979336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145702912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696735.979387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace92680 x1874531145702912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145702912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145702912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696735.979434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145703040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696735.979453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696735.979465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696735/real 1787696735] req@ffff8f51ace90000 x1874531145703040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696735.979495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145703040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696735.979505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696735.979508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696735.979511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145703040:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696736.874520:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696738.922521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696739.946572:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696740.970527:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696741.098665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145703168:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696741.098772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.098840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace91880 x1874531145703168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.098891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145703168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.098921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.098929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.098941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145703168:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696741.098955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696741.098981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.098994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace91c00 x1874531145703296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145703296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696741.099030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696741.099047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace91500 x1874531145703424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145703424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696741.099112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696741.099128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace92300 x1874531145703552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145703552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696741.099170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696741.099185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace92d80 x1874531145703680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145703680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145703680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696741.099227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145703808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696741.099241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace93480 x1874531145703808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145703808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145703808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696741.099284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145703936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696741.099299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace90000 x1874531145703936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145703936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145703936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696741.099348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145704064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696741.099421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace92680 x1874531145704064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145704064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145704064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696741.099468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145704192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696741.099485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696741.099495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696741/real 1787696741] req@ffff8f51ace90e00 x1874531145704192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696741.099506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145704192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696741.099522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696741.099525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696741.099528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145704192:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696743.018512:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696744.042519:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696745.066535:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696746.218469:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696746.218480:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696746.218485:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.218497:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218510:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218520:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696746.218574:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145704320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696746.218610:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696746.218614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696746.218616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.218620:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218624:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696746.218638:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696746.218660:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145704448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696746.218677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145704320:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696746.218687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696746.218693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696746.218695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.218700:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218704:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218710:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696746.218715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696746.218728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145704576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696746.218735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696746.218737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696746.218739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.218742:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218745:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218748:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696746.218754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696746.218754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696746.218772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145704704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696746.218782:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696746.218785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696746.218787:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.218791:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218797:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696746.218802:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696746.218816:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145704832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696746.218818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704c000 x1874531145704320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696746.218857:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696746.218868:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696746.218871:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.218878:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218881:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218884:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696746.218890:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145704320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696746.218890:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696746.218909:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145704960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696746.218914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696746.218929:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696746.218930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696746.218940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696746.218944:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696746.218945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145704320:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696746.218958:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.218970:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.218974:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696746.218979:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696746.218982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696746.218996:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145705088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696746.219011:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696746.219012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696746.219021:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696746.219023:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696746.219027:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00100000:1.0:1787696746.219034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704f800 x1874531145704448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696746.219035:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.219044:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696746.219056:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696746.219065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145704448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696746.219072:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145705216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696746.219106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696746.219118:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696746.219119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704448:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696746.219129:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696746.219132:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696746.219135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704576:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696746.219141:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4645 00000100:00080000:2.0:1787696746.219156:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696746.219162:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696746.219167:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696746.219173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696746.219194:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145705344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696746.219196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704c380 x1874531145704576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145704576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696746.219309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696746.219329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696746.219337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704e680 x1874531145704704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145704704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696746.219383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696746.219396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696746.219404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704ce00 x1874531145704832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145704832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145704832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696746.219425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145704960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696746.219436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696746.219442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704df80 x1874531145704960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145704960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145704960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696746.219472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145705088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696746.219482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696746.219488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704d880 x1874531145705088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145705088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145705088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696746.219513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145705216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696746.219523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696746.219529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704ea00 x1874531145705216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145705216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145705216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696746.219548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145705344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696746.219557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696746.219563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696746/real 1787696746] req@ffff8f518704c000 x1874531145705344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696746.219570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145705344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696746.219579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696746.219582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696746.219585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145705344:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696747.114485:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696748.138485:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696749.162494:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696751.210532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696751.338654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145705472:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696751.338775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696751.338822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696751.338870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696751.338913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696751.338946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696751.338984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696751.339038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696751.339077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706496:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696752.234597:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696753.258537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696754.410670:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696754.410719:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696754.410736:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410794:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410809:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410819:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410837:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410848:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410860:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410875:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696754.410893:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696754.410898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace90000 x1874531145706496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.410945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145706496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.410962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.410969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.410974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696754.410984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace92680 x1874531145706368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.410999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145706368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696754.411027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace90e00 x1874531145706240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145706240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696754.411071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace93b80 x1874531145705984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145705984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696754.411112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace93100 x1874531145706112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145706112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145706112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696754.411152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace90380 x1874531145705600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145705600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696754.411205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace93800 x1874531145705728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145705728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696754.411241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace90700 x1874531145705856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145705856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145705856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696754.411281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696751/real 1787696754] req@ffff8f51ace91180 x1874531145705472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696754.411297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145705472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696754.411307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696754.411311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696754.411315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145705472:192.168.201.125@tcp:250:kworker.0 00010000:00010000:1.0:1787696755.306506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696756.330588:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696756.458714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145706624:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696756.458798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.458852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace93480 x1874531145706624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.458884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145706624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.458903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.458912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.458921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145706624:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696756.458937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145706752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696756.458983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace92d80 x1874531145706752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145706752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145706752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696756.459075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145706880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696756.459103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace92300 x1874531145706880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145706880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145706880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696756.459220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145707008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696756.459244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace91500 x1874531145707008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145707008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145707008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696756.459289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145707136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696756.459309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace91c00 x1874531145707136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145707136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145707136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696756.459349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696756.459401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace91880 x1874531145707264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145707264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696756.459449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696756.459469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace91180 x1874531145707392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145707392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696756.459516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696756.459537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace90700 x1874531145707520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145707520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696756.459582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696756.459603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696756.459614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696756/real 1787696756] req@ffff8f51ace93800 x1874531145707648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696756.459633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145707648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696756.459641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696756.459644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696756.459647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145707648:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696757.354508:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696759.402543:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696760.426560:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696761.450539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696761.578585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145707776:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696761.578646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.578690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace90380 x1874531145707776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.578742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145707776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.578776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.578784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.578798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145707776:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696761.578830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145707904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696761.578872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.578895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace93100 x1874531145707904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.578928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145707904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.578939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.578944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.578949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145707904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696761.578958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145708032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696761.578978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.578990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace93b80 x1874531145708032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.578999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145708032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145708032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696761.579023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145708160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696761.579055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.579064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace90e00 x1874531145708160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.579072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145708160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145708160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696761.579086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145708288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696761.579097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.579103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace92680 x1874531145708288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.579111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145708288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145708288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696761.579123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696761.579135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.579145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace90000 x1874531145708416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.579157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145708416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696761.579179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696761.579200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.579210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace93800 x1874531145708544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.579223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145708544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696761.579259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696761.579291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.579310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace90700 x1874531145708672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.579320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145708672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696761.579333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696761.579347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696761.579382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696761/real 1787696761] req@ffff8f51ace91180 x1874531145708800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696761.579392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145708800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696761.579398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696761.579400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696761.579402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145708800:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696763.498584:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696764.522552:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696765.546629:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696766.698510:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698519:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696766.698522:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698531:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698541:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698547:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696766.698582:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145708928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698611:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698615:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696766.698617:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698628:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698641:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696766.698653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145709056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698666:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696766.698668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698672:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698675:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698677:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698680:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696766.698690:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145709184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696766.698702:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698709:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698712:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696766.698713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145708928:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696766.698716:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698720:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696766.698735:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145709312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698744:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696766.698745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698748:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698751:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698753:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698756:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696766.698763:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145709440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698774:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696766.698774:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698778:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698780:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698784:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696766.698798:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145709568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698812:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696766.698814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696766.698822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696766.698828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698835:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698840:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698846:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696766.698858:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145709696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698865:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698867:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696766.698869:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698873:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00080000:2.0:1787696766.698876:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698879:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698881:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696766.698887:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145709824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696766.698896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696766.698899:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696766.698901:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696766.698909:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4666 00000100:00100000:1.0:1787696766.698912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704ed80 x1874531145708928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696766.698913:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696766.698916:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696766.698919:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696766.698927:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145709952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696766.699003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145708928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145708928:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696766.699039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696766.699069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704dc00 x1874531145709056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145709056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696766.699146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696766.699168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704fb80 x1874531145709184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145709184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696766.699223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696766.699244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704d500 x1874531145709312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145709312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696766.699291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696766.699312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704f480 x1874531145709440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145709440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145709440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696766.699412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696766.699436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704e300 x1874531145709568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145709568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696766.699489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696766.699511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704d180 x1874531145709696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145709696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696766.699559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696766.699579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704c700 x1874531145709824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145709824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696766.699623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696766.699642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696766.699652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696766/real 1787696766] req@ffff8f518704c000 x1874531145709952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696766.699664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145709952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696766.699673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696766.699677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696766.699680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145709952:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696767.594549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696768.618528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696769.642528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696771.690549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696771.818643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145710080:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696771.818718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.818769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace91880 x1874531145710080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.818813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145710080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.818829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.818839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.818850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145710080:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696771.818909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696771.818961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.818995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace91c00 x1874531145710208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145710208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696771.819114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696771.819146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace91500 x1874531145710336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145710336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696771.819202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696771.819232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace92300 x1874531145710464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145710464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696771.819263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696771.819274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace92d80 x1874531145710592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145710592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145710592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696771.819325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145710720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696771.819391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace93480 x1874531145710720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145710720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145710720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696771.819445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145710848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696771.819460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace91880 x1874531145710848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145710848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145710848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696771.819488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145710976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696771.819507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace91180 x1874531145710976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145710976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145710976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696771.819534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145711104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696771.819543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696771.819549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696771/real 1787696771] req@ffff8f51ace90700 x1874531145711104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696771.819556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145711104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696771.819561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696771.819563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696771.819565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145711104:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696772.714522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696773.738541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696775.786524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696776.810535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696776.938532:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938544:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696776.938548:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938560:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938573:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938588:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696776.938632:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145711232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696776.938657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696776.938661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938665:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938668:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938670:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938680:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696776.938691:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145711360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696776.938705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696776.938708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145711232:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696776.938708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696776.938711:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938717:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938725:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696776.938739:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145711488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696776.938745:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696776.938750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938752:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938754:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938757:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938760:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696776.938767:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145711616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696776.938790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696776.938795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938798:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938802:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938805:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938810:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696776.938821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145711744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696776.938828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696776.938834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938838:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696776.938839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938843:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938845:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938848:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696776.938860:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145711872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696776.938866:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938869:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696776.938871:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938874:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938876:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938878:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938882:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696776.938891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704ea00 x1874531145711232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696776.938892:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145712000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696776.938944:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.938947:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696776.938948:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.938952:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.938954:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.938957:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.938960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696776.938967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145711232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696776.938970:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145712128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696776.938995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696776.938995:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696776.939003:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696776.939006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145711232:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696776.939011:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696776.939017:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4676 00000100:00080000:2.0:1787696776.939022:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696776.939027:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696776.939030:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696776.939037:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145712256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696776.939047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696776.939069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704d880 x1874531145711360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145711360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696776.939105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696776.939116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704df80 x1874531145711488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145711488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696776.939168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696776.939188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704ce00 x1874531145711616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145711616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696776.939219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696776.939230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704e680 x1874531145711744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145711744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145711744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696776.939257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145711872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696776.939268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704c380 x1874531145711872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145711872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145711872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696776.939294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145712000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696776.939304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704f800 x1874531145712000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145712000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145712000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696776.939330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145712128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696776.939339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704c000 x1874531145712128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145712128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145712128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696776.939414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145712256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696776.939428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696776.939435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696776/real 1787696776] req@ffff8f518704c700 x1874531145712256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696776.939443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145712256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696776.939449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696776.939451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696776.939453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145712256:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696777.834538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696779.882600:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696780.906624:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696781.930554:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696782.058546:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058557:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696782.058562:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058573:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.058586:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058593:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696782.058625:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145712384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696782.058652:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696782.058656:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058659:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.058661:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058664:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696782.058672:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696782.058697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145712512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696782.058710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058714:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696782.058717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058725:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00100000:1.0:1787696782.058732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145712384:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696782.058734:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058747:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696782.058752:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696782.058768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145712640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696782.058777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058781:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696782.058782:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058785:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.058789:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058791:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696782.058796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696782.058802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145712768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696782.058810:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058813:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696782.058814:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058816:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.058818:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058824:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696782.058831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696782.058834:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696782.058845:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145712896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696782.058852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058855:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696782.058857:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058860:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.058863:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058866:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696782.058871:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696782.058877:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145713024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696782.058884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058887:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696782.058888:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696782.058897:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00100000:1.0:1787696782.058898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f518738aa00 x1874531145712384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.058903:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696782.058919:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696782.058933:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696782.058950:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145713152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696782.058961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145712384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.058963:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696782.058985:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787696782.058986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.058995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696782.058995:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696782.059002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145712384:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696782.059008:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.059013:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696782.059015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712512:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696782.059021:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696782.059027:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696782.059043:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145713280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696782.059051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696782.059066:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696782.059067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f518738b480 x1874531145712512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.059090:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696782.059096:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696782.059107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145712512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696782.059112:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4681 00000100:00080000:3.0:1787696782.059124:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696782.059130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696782.059133:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696782.059139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787696782.059142:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696782.059145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696782.059167:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145713408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696782.059177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696782.059203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f518738ad80 x1874531145712640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145712640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696782.059242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696782.059254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f518738bb80 x1874531145712768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145712768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696782.059281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696782.059292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f518738b800 x1874531145712896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145712896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145712896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696782.059319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696782.059331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f5187388000 x1874531145713024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145713024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696782.059408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696782.059426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f5187388380 x1874531145713152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145713152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696782.059458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696782.059469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f5187389c00 x1874531145713280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145713280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696782.059508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696782.059518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696782.059525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696782/real 1787696782] req@ffff8f518738b100 x1874531145713408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696782.059532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145713408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696782.059546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696782.059548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696782.059551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145713408:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696783.978516:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696785.002540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696786.026518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696787.178417:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178426:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696787.178429:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178438:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178448:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178453:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696787.178486:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145713536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178510:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178513:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696787.178514:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178517:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178520:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178522:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178529:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696787.178538:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145713664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178546:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178548:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696787.178550:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178552:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178555:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178558:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178561:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696787.178569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145713792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178577:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178580:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696787.178581:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178583:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178585:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178604:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178609:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696787.178619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145713536:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696787.178619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145713920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696787.178644:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178649:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178653:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178658:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696787.178675:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145714048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178690:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696787.178692:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178696:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178706:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178713:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696787.178718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696787.178734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145714176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178742:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178746:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696787.178748:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178751:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178755:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178757:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178763:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696787.178769:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145714304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178781:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696787.178787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f518738a300 x1874531145713536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696787.178788:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696787.178794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178812:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178818:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178824:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696787.178827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696787.178840:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145714432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696787.178847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696787.178851:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696787.178858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145713536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696787.178858:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696787.178876:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4686 00000100:00080000:3.0:1787696787.178885:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696787.178893:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696787.178894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787696787.178907:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696787.178908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.178920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145713536:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696787.178928:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145714560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696787.178936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145713664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696787.178966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.178983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f5187389f80 x1874531145713664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.178997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145713664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145713664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696787.179045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145713792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696787.179063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f5187388a80 x1874531145713792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145713792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145713792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696787.179118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145713920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696787.179138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f518738a680 x1874531145713920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145713920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145713920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696787.179170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145714048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696787.179182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f5187388e00 x1874531145714048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145714048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145714048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696787.179210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696787.179221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f5187389180 x1874531145714176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145714176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696787.179248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696787.179259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f5187389500 x1874531145714304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145714304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696787.179285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696787.179296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f518738b100 x1874531145714432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145714432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696787.179322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696787.179333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696787.179339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696787/real 1787696787] req@ffff8f5187389c00 x1874531145714560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696787.179346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145714560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696787.179351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696787.179393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696787.179396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145714560:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696788.074470:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696789.098672:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696790.122483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696792.170552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696792.298708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145714688:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696792.298829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.298913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace93800 x1874531145714688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.298956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145714688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.298968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.298973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.298977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145714688:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696792.298988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145714816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696792.299025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace90000 x1874531145714816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145714816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145714816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696792.299093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145714944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696792.299108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace92680 x1874531145714944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145714944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145714944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696792.299143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145715072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696792.299155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace90e00 x1874531145715072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145715072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145715072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696792.299182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145715200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696792.299192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace93b80 x1874531145715200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145715200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145715200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696792.299220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696792.299231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace93100 x1874531145715328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145715328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696792.299257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696792.299268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace90380 x1874531145715456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145715456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696792.299294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696792.299304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace90700 x1874531145715584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145715584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696792.299330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696792.299340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696792.299345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696792/real 1787696792] req@ffff8f51ace91180 x1874531145715712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696792.299390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145715712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696792.299396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696792.299398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696792.299400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145715712:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696793.194533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696794.218505:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696796.266483:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696797.290518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696797.418450:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418462:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696797.418467:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418477:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418489:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418497:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696797.418528:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145715840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696797.418551:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418556:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696797.418558:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418574:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418585:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696797.418596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145715840:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696797.418603:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145715968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696797.418617:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696797.418623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418629:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418633:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418637:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418643:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696797.418654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696797.418658:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145716096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696797.418683:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418686:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696797.418687:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696797.418693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f5187388380 x1874531145715840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696797.418694:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418707:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418715:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696797.418727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145716224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696797.418736:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418739:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696797.418740:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418744:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00100000:1.0:1787696797.418746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145715840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696797.418747:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418752:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418758:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696797.418769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696797.418775:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145716352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696797.418777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.418785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145715840:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696797.418796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418799:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696797.418800:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418804:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418807:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418810:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696797.418815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145715968:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787696797.418816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696797.418837:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145716480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696797.418845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696797.418851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418855:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696797.418856:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696797.418862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f5187388000 x1874531145715968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696797.418862:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418865:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418871:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418878:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696797.418884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145715968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696797.418890:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145716608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696797.418895:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696797.418904:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418906:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696797.418907:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418910:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:1.0:1787696797.418913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696797.418914:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418918:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418923:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696797.418928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145715968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696797.418934:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145716736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696797.418943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145716096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696797.418960:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696797.418962:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696797.418964:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696797.418966:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4697 00000100:00080000:3.0:1787696797.418969:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696797.418971:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696797.418974:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696797.418982:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145716864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696797.418988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f518738b800 x1874531145716096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145716096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145716096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696797.419035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145716224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696797.419048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f518738bb80 x1874531145716224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145716224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145716224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696797.419086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145716352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696797.419097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f518738ad80 x1874531145716352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145716352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145716352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696797.419133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696797.419144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f518738b480 x1874531145716480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145716480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696797.419170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696797.419179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f518738aa00 x1874531145716608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145716608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696797.419204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696797.419212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f5187388380 x1874531145716736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145716736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696797.419237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696797.419246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696797.419251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696797/real 1787696797] req@ffff8f5187389c00 x1874531145716864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696797.419257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145716864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696797.419262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696797.419264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696797.419266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145716864:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696798.314512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696800.362579:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696801.386531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696802.410603:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696802.538697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145716992:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696802.538790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.538854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace91880 x1874531145716992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.538900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145716992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.538921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.538930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.538942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145716992:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696802.538983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696802.539032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace93480 x1874531145717120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145717120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696802.539086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696802.539104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace92d80 x1874531145717248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145717248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696802.539158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696802.539178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace92300 x1874531145717376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145717376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696802.539209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696802.539220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace91500 x1874531145717504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145717504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145717504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696802.539247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145717632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696802.539258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace91c00 x1874531145717632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145717632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145717632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696802.539283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145717760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696802.539294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace91180 x1874531145717760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145717760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145717760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696802.539350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145717888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696802.539397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace90700 x1874531145717888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145717888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145717888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696802.539427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145718016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696802.539438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696802.539444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696802/real 1787696802] req@ffff8f51ace90380 x1874531145718016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696802.539451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145718016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696802.539456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696802.539458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696802.539460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145718016:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696804.458545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696805.482491:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696806.506546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696807.658566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145718144:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696807.658618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.658655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace93100 x1874531145718144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.658708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145718144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.658729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.658737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.658743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145718144:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696807.658767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696807.658789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.658802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace93b80 x1874531145718272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.658830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145718272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.658843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.658850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.658856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696807.658865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696807.658879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.658887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace90e00 x1874531145718400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.658895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145718400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.658903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.658905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.658907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696807.658910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696807.658919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.658924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace92680 x1874531145718528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.658930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145718528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.658934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.658936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.658937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696807.658940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696807.658948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.658953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace90000 x1874531145718656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.658959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145718656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.658963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.658965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.658967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145718656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696807.658970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145718784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696807.658983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.658993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace93800 x1874531145718784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.659003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145718784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.659011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.659013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.659015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145718784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696807.659026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145718912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696807.659038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.659045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace90380 x1874531145718912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.659052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145718912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.659056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.659058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.659059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145718912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696807.659062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145719040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696807.659070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.659075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace90700 x1874531145719040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.659081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145719040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.659085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.659087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.659088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145719040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696807.659091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145719168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696807.659102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696807.659111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696807/real 1787696807] req@ffff8f51ace91180 x1874531145719168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696807.659121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145719168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696807.659129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696807.659132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696807.659135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145719168:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696808.554563:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696809.578513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696810.602665:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696812.650520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696812.780198:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780209:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696812.780213:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780229:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780240:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780247:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696812.780283:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145719296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780314:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780319:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696812.780321:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780325:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780329:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780333:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696812.780386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145719296:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787696812.780386:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696812.780405:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145719424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780417:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780421:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696812.780423:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780427:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780431:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780435:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780440:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696812.780451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696812.780453:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145719552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780470:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780474:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696812.780476:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780480:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780483:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780487:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780491:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696812.780501:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145719680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780509:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780512:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696812.780514:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780517:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780520:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780523:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780528:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696812.780540:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145719808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780552:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696812.780553:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780560:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00100000:1.0:1787696812.780561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704d180 x1874531145719296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696812.780564:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780571:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780577:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696812.780589:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145719936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780609:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696812.780613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696812.780614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145719296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696812.780618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780621:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696812.780638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145720064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780648:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696812.780650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780653:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780655:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780659:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780663:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696812.780672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145720192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696812.780680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696812.780682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696812.780685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696812.780688:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4712 00000100:00080000:2.0:1787696812.780691:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696812.780694:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696812.780699:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696812.780708:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145720320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696812.780729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.780734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.780739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145719296:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696812.780761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696812.780785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.780798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704e300 x1874531145719424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.780810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145719424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.780829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.780833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.780836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696812.780842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696812.780862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.780873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704f480 x1874531145719552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.780884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145719552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.780897:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.780900:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.780903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696812.780909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696812.780932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.780943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704d500 x1874531145719680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.780954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145719680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.780962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.780965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.780968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696812.780974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696812.780991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.781001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704fb80 x1874531145719808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.781012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145719808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.781019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.781022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.781025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145719808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696812.781030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145719936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696812.781047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.781057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704dc00 x1874531145719936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.781068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145719936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.781076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.781079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.781082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145719936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696812.781087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145720064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696812.781104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.781115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704ed80 x1874531145720064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.781127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145720064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.781133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.781136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.781139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145720064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696812.781145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145720192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696812.781177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.781190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704c700 x1874531145720192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.781201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145720192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.781208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.781210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.781213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145720192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696812.781219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145720320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696812.781236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696812.781247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696812/real 1787696812] req@ffff8f518704c000 x1874531145720320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696812.781258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145720320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696812.781265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696812.781268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696812.781272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145720320:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696813.674597:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696814.698602:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696816.746563:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696817.770537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696817.898502:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898510:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696817.898514:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898523:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898532:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898539:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696817.898579:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145720448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696817.898616:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898620:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898623:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898626:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696817.898652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145720576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898662:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696817.898667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898670:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898673:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898677:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696817.898693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145720704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898705:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696817.898706:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898713:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898717:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898721:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696817.898732:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145720832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898740:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696817.898745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898748:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898751:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898755:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898759:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696817.898769:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145720960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898777:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696817.898782:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898785:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898789:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898792:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898796:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696817.898804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145721088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898812:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898815:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696817.898817:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898821:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898824:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898827:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696817.898842:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145721216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898853:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696817.898855:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898858:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898861:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898865:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898869:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696817.898880:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145721344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696817.898889:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696817.898892:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696817.898894:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696817.898897:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4717 00000100:00080000:3.0:1787696817.898900:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696817.898904:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696817.898908:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696817.898918:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145721472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696817.899008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145720448:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696817.899133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696817.899161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696817.899182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696817.899204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696817.899223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696817.899243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696817.899264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696817.899284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721472:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696818.794535:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696820.842535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696820.970690:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696820.970733:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696820.970750:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970797:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970810:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970818:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970825:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970833:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970841:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970849:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696820.970855:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696820.970979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f518738b100 x1874531145720448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145720448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145720448:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696820.971060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f5187389500 x1874531145720576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145720576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696820.971110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f5187389180 x1874531145720704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145720704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696820.971140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f5187388e00 x1874531145720832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145720832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696820.971170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f518738a680 x1874531145720960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145720960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145720960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696820.971201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f5187388a80 x1874531145721088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145721088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696820.971231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f5187389f80 x1874531145721216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145721216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696820.971258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f518738a300 x1874531145721344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145721344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696820.971289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696817/real 1787696820] req@ffff8f5187389c00 x1874531145721472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696820.971300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145721472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696820.971307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696820.971310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696820.971313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145721472:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696821.866602:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696822.890528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696823.018498:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018509:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696823.018513:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018523:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018535:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018542:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696823.018579:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145721600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018608:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018612:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696823.018614:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018619:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018623:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018628:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018639:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696823.018653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145721728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018666:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696823.018667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018671:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018676:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018679:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018684:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696823.018696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145721856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018704:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696823.018708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018712:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018715:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018719:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696823.018734:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145721984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018750:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018754:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696823.018755:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018759:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018763:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018766:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696823.018782:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145722112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696823.018794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145721600:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696823.018797:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018813:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696823.018815:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018821:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018825:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018831:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018837:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696823.018852:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145722240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018864:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696823.018866:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018869:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018873:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018877:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018881:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696823.018891:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145722368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018900:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018903:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696823.018904:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018908:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018911:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696823.018914:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018919:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696823.018931:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145722496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696823.018940:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696823.018944:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696823.018946:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696823.018951:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4722 00000100:00080000:3.0:1787696823.018956:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696823.018963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696823.018965:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696823.018990:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696823.019008:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145722624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696823.019043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f5187388380 x1874531145721600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145721600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145721600:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696823.019094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145721728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696823.019117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f518738aa00 x1874531145721728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145721728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145721728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696823.019178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145721856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696823.019198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f518738b480 x1874531145721856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145721856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145721856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696823.019240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145721984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696823.019259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f518738ad80 x1874531145721984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145721984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145721984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696823.019328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145722112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696823.019346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f518738bb80 x1874531145722112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145722112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145722112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696823.019442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696823.019471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f518738b800 x1874531145722240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145722240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696823.019517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696823.019536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f5187388000 x1874531145722368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145722368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696823.019580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696823.019598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f5187389c00 x1874531145722496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145722496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696823.019640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696823.019659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696823.019671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696823/real 1787696823] req@ffff8f518738a300 x1874531145722624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696823.019682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145722624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696823.019690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696823.019693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696823.019696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145722624:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696824.938511:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696825.962550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696826.986523:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696828.138409:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138419:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696828.138423:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138432:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138441:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138449:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696828.138485:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145722752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696828.138510:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138513:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696828.138514:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138518:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138521:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138524:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138531:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696828.138541:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145722880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696828.138553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145722752:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696828.138553:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138562:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696828.138565:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138570:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138574:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138578:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138583:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696828.138596:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145723008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696828.138602:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138604:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696828.138605:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138607:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138611:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696828.138616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696828.138618:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696828.138630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145723136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696828.138636:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696828.138639:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138641:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138643:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138645:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138648:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696828.138653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145723264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696828.138659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138662:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696828.138663:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696828.138667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f5187389f80 x1874531145722752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696828.138670:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138676:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138684:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138691:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696828.138698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145722752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696828.138705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145723392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696828.138713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696828.138717:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696828.138721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696828.138721:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696828.138725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696828.138731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145722752:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696828.138731:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:3.0:1787696828.138734:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138742:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138746:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696828.138758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145723520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696828.138760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145722880:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696828.138777:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696828.138780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696828.138782:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138787:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4727 00000800:00000100:1.0:1787696828.138792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696828.138794:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696828.138800:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696828.138805:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696828.138808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f5187388a80 x1874531145722880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696828.138818:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145723648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696828.138842:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696828.138844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145722880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696828.138846:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696828.138849:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696828.138858:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4727 00000100:00080000:1.0:1787696828.138863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696828.138864:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696828.138872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696828.138873:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696828.138879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145722880:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787696828.138881:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696828.138909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145723776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696828.138917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145723008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696828.138940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.138950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f518738a680 x1874531145723008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.138961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145723008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.138968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.138972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.138975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145723008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696828.138979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145723136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696828.138992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.138999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f5187388e00 x1874531145723136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.139010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145723136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.139023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.139025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.139028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145723136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696828.139033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145723264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696828.139044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.139052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f5187389180 x1874531145723264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.139061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145723264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.139068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.139071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.139074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145723264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696828.139079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696828.139091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.139098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f5187389500 x1874531145723392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.139109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145723392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.139116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.139119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.139122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696828.139136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696828.139149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.139156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f518738b100 x1874531145723520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.139167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145723520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.139174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.139177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.139179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696828.139184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696828.139195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.139202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f5187389f80 x1874531145723648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.139212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145723648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.139219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.139222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.139224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696828.139229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696828.139240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696828.139248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696828/real 1787696828] req@ffff8f518738a300 x1874531145723776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696828.139258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145723776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696828.139265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696828.139267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696828.139270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145723776:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696829.034508:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696830.058559:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696831.082502:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696833.130524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696833.258504:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.258517:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696833.258521:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696833.258532:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.258544:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.258552:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696833.258607:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145723904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696833.258639:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.258643:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696833.258646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696833.258651:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.258654:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.258659:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696833.258670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696833.258683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145723904:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696833.258699:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145724032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696833.258725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.258728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696833.258730:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696833.258735:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.258739:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.258743:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696833.258749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696833.258749:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696833.258768:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145724160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696833.258780:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.258784:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696833.258787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704f800 x1874531145723904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696833.258788:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696833.258813:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.258821:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.258827:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696833.258831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145723904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696833.258835:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696833.258847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.258850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.258857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145723904:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696833.258859:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145724288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696833.258872:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.258876:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696833.258880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724032:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696833.258880:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696833.258890:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.258895:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.258899:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696833.258906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696833.258906:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696833.258919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704c380 x1874531145724032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696833.258926:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145724416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696833.258962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145724032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696833.258962:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.258972:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787696833.258979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.258982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.258985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724032:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696833.258987:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696833.258998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724160:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696833.258998:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.259005:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.259012:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696833.259018:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696833.259030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696833.259037:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145724544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696833.259046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704e680 x1874531145724160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696833.259052:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.259061:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696833.259064:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696833.259069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145724160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696833.259073:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.259080:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696833.259083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696833.259087:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696833.259089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724160:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787696833.259096:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696833.259102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696833.259112:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145724672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696833.259123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696833.259134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704ce00 x1874531145724288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696833.259136:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.259146:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696833.259151:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696833.259156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145724288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696833.259158:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.259165:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696833.259172:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696833.259180:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696833.259192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787696833.259197:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145724800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696833.259200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724288:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696833.259217:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696833.259222:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696833.259225:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696833.259229:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4732 00000100:00080000:2.0:1787696833.259233:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696833.259235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724416:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696833.259241:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696833.259247:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696833.259260:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145724928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696833.259270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696833.259290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704df80 x1874531145724416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696833.259304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145724416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696833.259312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.259315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145724416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696833.259325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696833.259346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696833.259388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704d880 x1874531145724544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696833.259402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145724544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696833.259412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.259415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696833.259426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696833.259447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696833.259460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704ea00 x1874531145724672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696833.259473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145724672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696833.259481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.259484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696833.259496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696833.259517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696833.259529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704f800 x1874531145724800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696833.259542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145724800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696833.259551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.259554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696833.259575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696833.259593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696833.259603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696833/real 1787696833] req@ffff8f518704c000 x1874531145724928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696833.259611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145724928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696833.259617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696833.259618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696833.259620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145724928:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696834.154564:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696835.178524:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696837.226567:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696838.250491:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696838.378592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145725056:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696838.378688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.378739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace91c00 x1874531145725056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.378762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145725056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.378781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.378787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.378794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145725056:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696838.378833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696838.378876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.378886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace91500 x1874531145725184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.378914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145725184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.378924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.378927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.378930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696838.378954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696838.378976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.378989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace92300 x1874531145725312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145725312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696838.379024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696838.379043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.379052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace92d80 x1874531145725440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145725440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696838.379090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696838.379111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.379123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace93480 x1874531145725568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145725568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145725568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696838.379146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145725696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696838.379159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.379166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace91880 x1874531145725696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145725696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145725696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696838.379188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145725824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696838.379197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.379203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace91180 x1874531145725824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145725824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145725824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696838.379222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145725952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696838.379231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.379237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace90700 x1874531145725952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145725952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145725952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696838.379255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145726080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696838.379264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696838.379270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696838/real 1787696838] req@ffff8f51ace90380 x1874531145726080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696838.379278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145726080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696838.379283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696838.379284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696838.379286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145726080:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696839.274550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696841.322548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696842.346495:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696843.370563:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696843.498648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145726208:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696843.498724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.498771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace93800 x1874531145726208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.498800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145726208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.498818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.498826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.498834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145726208:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696843.498865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696843.498905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.498925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace90000 x1874531145726336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.498963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145726336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.498994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696843.499023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696843.499052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace92680 x1874531145726464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145726464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696843.499141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696843.499173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace90e00 x1874531145726592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145726592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696843.499225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696843.499253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace93b80 x1874531145726720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145726720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145726720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696843.499306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145726848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696843.499323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace93100 x1874531145726848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145726848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145726848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696843.499392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145726976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696843.499409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace93800 x1874531145726976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145726976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145726976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696843.499452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145727104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696843.499468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace90380 x1874531145727104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145727104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145727104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696843.499521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145727232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696843.499537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696843.499547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696843/real 1787696843] req@ffff8f51ace90700 x1874531145727232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696843.499558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145727232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696843.499566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696843.499569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696843.499573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145727232:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696845.418561:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696846.442635:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696847.466630:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696848.618481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618490:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696848.618494:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618502:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618512:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618518:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696848.618564:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145727360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696848.618591:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696848.618597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618602:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618606:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618611:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618621:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696848.618639:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145727488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696848.618641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145727360:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696848.618656:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696848.618660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618663:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618666:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618669:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618673:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696848.618684:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145727616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696848.618690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696848.618694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618697:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618702:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618706:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696848.618710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696848.618716:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145727744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696848.618723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618726:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696848.618727:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618730:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618732:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618734:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618737:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696848.618744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145727872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696848.618751:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618753:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696848.618755:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696848.618757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f5187389c00 x1874531145727360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696848.618761:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618766:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618773:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618778:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696848.618782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145727360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696848.618788:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145728000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696848.618796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.618804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.618811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145727360:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696848.618811:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696848.618820:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618824:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618826:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696848.618828:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696848.618838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696848.618840:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145728128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696848.618849:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618851:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696848.618853:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618856:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:3.0:1787696848.618859:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696848.618863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696848.618866:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696848.618870:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696848.618877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f5187388000 x1874531145727488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696848.618883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145728256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696848.618907:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696848.618921:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696848.618923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145727488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696848.618929:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696848.618938:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4748 00000100:00080000:1.0:1787696848.618939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696848.618944:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696848.618947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696848.618951:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696848.618956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727488:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787696848.618957:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696848.618968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696848.618974:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145728384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696848.618994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f518738b800 x1874531145727616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145727616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696848.619044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696848.619065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f518738bb80 x1874531145727744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145727744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696848.619111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696848.619126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f518738ad80 x1874531145727872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145727872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145727872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696848.619152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696848.619162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f518738b480 x1874531145728000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145728000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696848.619198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696848.619208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f518738aa00 x1874531145728128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145728128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696848.619233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696848.619242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f5187388380 x1874531145728256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145728256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696848.619265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696848.619274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696848.619280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696848/real 1787696848] req@ffff8f5187389c00 x1874531145728384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696848.619287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145728384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696848.619292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696848.619294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696848.619295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145728384:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696849.514675:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696850.538554:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696851.562566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696853.610637:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696853.738675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145728512:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696853.738741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.738780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace91180 x1874531145728512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.738803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145728512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.738813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.738819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.738826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145728512:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696853.738851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145728640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696853.738876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.738894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace91880 x1874531145728640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.738943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145728640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.738955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.738962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.738969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145728640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696853.738977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145728768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696853.739011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace93480 x1874531145728768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145728768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145728768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696853.739099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145728896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696853.739135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace92d80 x1874531145728896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145728896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145728896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696853.739228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145729024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696853.739246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace92300 x1874531145729024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145729024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145729024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696853.739276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696853.739286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace91500 x1874531145729152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145729152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696853.739311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696853.739320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace91c00 x1874531145729280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145729280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696853.739345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696853.739382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace91180 x1874531145729408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145729408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696853.739410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696853.739422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696853.739429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696853/real 1787696853] req@ffff8f51ace90700 x1874531145729536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696853.739436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145729536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696853.739441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696853.739460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696853.739463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145729536:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696854.634629:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696855.658546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696857.706531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696858.730557:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696858.858462:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696858.858471:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696858.858475:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858484:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858495:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.858502:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696858.858540:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145729664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696858.858564:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696858.858567:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696858.858568:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858571:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858574:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.858577:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696858.858586:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696858.858596:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145729792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696858.858609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696858.858610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145729664:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696858.858618:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696858.858621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858627:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858632:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.858636:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696858.858642:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696858.858658:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145729920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696858.858669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696858.858670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696858.858677:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696858.858680:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858686:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858691:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.858695:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696858.858701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696858.858709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704c700 x1874531145729664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696858.858716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145730048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696858.858729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145729664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696858.858730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696858.858740:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696858.858744:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696858.858751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696858.858752:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858760:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696858.858761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696858.858768:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696858.858770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145729664:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787696858.858774:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696858.858788:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145730176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696858.858795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696858.858798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787696858.858800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145729792:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696858.858801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858817:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858822:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696858.858828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696858.858829:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696858.858836:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696858.858844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704ed80 x1874531145729792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696858.858849:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145730304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696858.858871:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696858.858877:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696858.858878:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858882:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858885:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.858891:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696858.858895:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696858.858898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145729792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696858.858908:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145730432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696858.858914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.858922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696858.858927:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696858.858928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145729792:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696858.858937:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696858.858939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145729920:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696858.858941:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696858.858948:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.858953:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.858957:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696858.858962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696858.858963:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696858.858975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704dc00 x1874531145729920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696858.858981:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145730560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696858.858988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145729920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696858.859000:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696858.859002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696858.859009:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696858.859013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145729920:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696858.859014:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696858.859020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145730048:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696858.859023:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4758 00000100:00080000:2.0:1787696858.859029:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696858.859033:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696858.859038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696858.859039:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696858.859048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704fb80 x1874531145730048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696858.859056:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145730688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696858.859061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145730048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696858.859088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696858.859097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145730048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696858.859105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145730176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696858.859126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696858.859138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704d500 x1874531145730176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696858.859147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145730176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696858.859152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696858.859156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145730176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696858.859170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696858.859183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696858.859190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704f480 x1874531145730304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696858.859199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145730304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696858.859208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696858.859215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696858.859221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696858.859240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696858.859252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704e300 x1874531145730432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696858.859261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145730432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696858.859266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696858.859271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696858.859275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696858.859287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696858.859294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704d180 x1874531145730560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696858.859301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145730560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696858.859306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696858.859310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696858.859313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696858.859322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696858.859328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696858/real 1787696858] req@ffff8f518704c700 x1874531145730688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696858.859335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145730688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696858.859340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696858.859342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696858.859344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145730688:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696859.754520:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696861.802503:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696862.826552:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696863.850574:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696863.978430:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978439:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696863.978444:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978455:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978467:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978475:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696863.978528:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145730816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696863.978567:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978572:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696863.978574:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978579:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978584:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978588:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978599:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696863.978615:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145730944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696863.978624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145730816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696863.978631:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978636:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696863.978638:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978643:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978646:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978649:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978653:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696863.978665:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145731072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696863.978672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978675:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696863.978676:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978678:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978681:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978683:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978694:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696863.978696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696863.978705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145731200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696863.978713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696863.978717:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978720:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978723:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978725:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978728:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696863.978734:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145731328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696863.978740:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696863.978744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978747:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978749:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978751:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696863.978759:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145731456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696863.978765:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696863.978771:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696863.978773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f518738a300 x1874531145730816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696863.978778:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978787:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978794:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978799:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696863.978802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145730816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.978814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696863.978815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145731584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696863.978821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.978828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145730816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696863.978842:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696863.978846:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978851:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00080000:3.0:1787696863.978854:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696863.978855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145730944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696863.978861:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696863.978878:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145731712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696863.978891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696863.978896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696863.978900:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696863.978904:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696863.978908:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4763 00000100:00100000:1.0:1787696863.978910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f5187389f80 x1874531145730944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696863.978912:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696863.978921:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696863.978935:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696863.978939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145730944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.978952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696863.978953:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145731840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696863.978970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.978976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145730944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696863.978983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145731072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696863.979006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f518738b100 x1874531145731072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145731072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145731072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696863.979051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145731200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696863.979071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f5187389500 x1874531145731200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145731200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145731200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696863.979109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145731328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696863.979126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f5187389180 x1874531145731328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145731328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145731328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696863.979171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696863.979188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f5187388e00 x1874531145731456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145731456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696863.979228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696863.979244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f518738a680 x1874531145731584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145731584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696863.979305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696863.979319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f5187388a80 x1874531145731712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145731712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696863.979382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696863.979407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696863.979419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696863/real 1787696863] req@ffff8f518738a300 x1874531145731840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696863.979427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145731840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696863.979434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696863.979437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696863.979440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145731840:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696865.898569:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696866.922552:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696867.946551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696869.098475:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098483:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696869.098487:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098495:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098504:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098510:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696869.098554:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145731968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696869.098575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696869.098579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098583:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098585:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098588:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696869.098605:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145732096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696869.098612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145731968:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696869.098630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098635:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696869.098637:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098643:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098647:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098652:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696869.098672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696869.098676:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145732224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696869.098689:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098691:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696869.098693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098696:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098703:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696869.098715:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145732352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696869.098722:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098725:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696869.098728:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696869.098730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704c000 x1874531145731968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696869.098733:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098737:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098744:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098751:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696869.098760:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145732480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696869.098764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145731968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696869.098775:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098781:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696869.098783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696869.098784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696869.098788:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:1.0:1787696869.098793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696869.098793:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696869.098802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145731968:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696869.098802:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098811:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696869.098820:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145732608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696869.098825:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696869.098828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787696869.098831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696869.098832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098840:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098844:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098846:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696869.098858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145732736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696869.098864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696869.098876:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696869.098879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704f800 x1874531145732096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696869.098880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696869.098883:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098891:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00080000:2.0:1787696869.098895:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098901:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098908:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696869.098914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145732096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696869.098918:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145732864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696869.098931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696869.098935:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696869.098938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696869.098940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696869.098944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696869.098945:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696869.098954:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4768 00000100:00100000:1.0:1787696869.098957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732224:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696869.098958:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696869.098965:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696869.098969:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696869.098978:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145732992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696869.098984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704ea00 x1874531145732224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145732224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696869.099044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696869.099062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704d880 x1874531145732352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145732352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696869.099097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696869.099107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704df80 x1874531145732480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145732480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145732480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696869.099143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696869.099164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704ce00 x1874531145732608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145732608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696869.099228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696869.099243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704e680 x1874531145732736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145732736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696869.099270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696869.099280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704c380 x1874531145732864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145732864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696869.099310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696869.099320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696869.099326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696869/real 1787696869] req@ffff8f518704c000 x1874531145732992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696869.099333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145732992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696869.099339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696869.099341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696869.099349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145732992:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696869.994497:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696871.018553:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696872.042532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696874.090564:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696874.218486:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.218498:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696874.218503:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696874.218523:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.218536:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696874.218544:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696874.218585:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145733120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696874.218627:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.218632:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696874.218634:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696874.218640:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.218644:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696874.218648:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696874.218659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696874.218674:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145733248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696874.218686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.218690:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696874.218693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696874.218698:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.218702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696874.218710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145733120:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696874.218713:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696874.218738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696874.218765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145733376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696874.218778:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.218782:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696874.218784:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696874.218789:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.218794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696874.218798:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696874.218805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696874.218808:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696874.218830:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145733504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696874.218843:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.218847:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696874.218849:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696874.218854:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00100000:1.0:1787696874.218866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704c700 x1874531145733120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696874.218868:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696874.218884:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696874.218893:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696874.218900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145733120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696874.218911:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145733632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696874.218912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.218922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.218930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145733120:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696874.218946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696874.218954:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.218960:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696874.218963:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696874.218973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696874.218973:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.218984:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696874.218989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704d180 x1874531145733248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696874.218996:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696874.219013:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696874.219018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145733248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696874.219027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696874.219040:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145733760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696874.219055:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.219059:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696874.219062:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696874.219064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733376:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696874.219071:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.219079:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696874.219089:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696874.219103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696874.219116:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696874.219119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704e300 x1874531145733376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696874.219135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145733376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696874.219144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787696874.219144:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145733888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696874.219148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696874.219156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696874.219174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696874.219176:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.219182:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787696874.219184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704f480 x1874531145733504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696874.219187:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696874.219200:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00100000:1.0:1787696874.219206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145733504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696874.219209:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696874.219217:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696874.219227:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696874.219242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696874.219258:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145734016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696874.219261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733632:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696874.219284:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696874.219288:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696874.219292:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696874.219301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696874.219302:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4773 00000100:00080000:2.0:1787696874.219310:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696874.219317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704d500 x1874531145733632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696874.219318:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696874.219330:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696874.219352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145733632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696874.219393:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145734144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696874.219398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145733632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696874.219431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145733760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696874.219451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696874.219459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704fb80 x1874531145733760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696874.219474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145733760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696874.219480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145733760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696874.219498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145733888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696874.219510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696874.219516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704dc00 x1874531145733888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696874.219523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145733888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696874.219528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145733888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696874.219535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145734016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696874.219544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696874.219550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704ed80 x1874531145734016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696874.219556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145734016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696874.219561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145734016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696874.219567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145734144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696874.219575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696874.219581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696874/real 1787696874] req@ffff8f518704d180 x1874531145734144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696874.219588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145734144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696874.219593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696874.219594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696874.219596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145734144:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696875.114558:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696876.138574:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696878.186549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696879.210572:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696879.338712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145734272:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696879.338786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.338838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace90380 x1874531145734272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.338868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145734272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.338887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.338893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.338897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145734272:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696879.338923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696879.338946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.338960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace93800 x1874531145734400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145734400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696879.339061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696879.339085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace93100 x1874531145734528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145734528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696879.339161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696879.339198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace93b80 x1874531145734656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145734656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696879.339265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696879.339283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace90e00 x1874531145734784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145734784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145734784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696879.339319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145734912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696879.339331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace92680 x1874531145734912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145734912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145734912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696879.339381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145735040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696879.339394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace90000 x1874531145735040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145735040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145735040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696879.339422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145735168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696879.339432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace90380 x1874531145735168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145735168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145735168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696879.339457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145735296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696879.339466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696879.339475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696879/real 1787696879] req@ffff8f51ace90700 x1874531145735296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696879.339488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145735296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696879.339496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696879.339499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696879.339509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145735296:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696880.234640:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696882.282583:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696883.306513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696884.330512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696884.458463:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458473:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696884.458477:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458487:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458496:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458502:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696884.458535:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145735424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696884.458559:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458563:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696884.458565:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458569:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458572:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458576:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458591:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696884.458599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145735424:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696884.458605:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145735552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696884.458623:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458627:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696884.458629:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458633:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458637:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458641:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696884.458658:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145735680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696884.458667:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458670:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696884.458672:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458676:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458679:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458683:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458688:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696884.458701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145735808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696884.458706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696884.458716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458720:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696884.458722:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458727:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458731:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458735:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696884.458742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696884.458756:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145735936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696884.458768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735808:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787696884.458768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696884.458778:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458783:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458787:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458791:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696884.458800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696884.458811:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145736064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696884.458824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458827:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696884.458830:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696884.458835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736064:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696884.458835:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458841:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458846:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458851:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696884.458863:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145736192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696884.458884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458887:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696884.458889:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458892:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458896:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458900:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696884.458910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696884.458927:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145736320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696884.458936:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696884.458941:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696884.458943:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696884.458947:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4784 00000100:00080000:2.0:1787696884.458950:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696884.458954:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696884.458959:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696884.458970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787696884.458972:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145736448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696884.458996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736448:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696886.378613:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696887.402478:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696887.530637:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696887.530675:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696887.530690:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530735:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530745:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530752:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530762:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530771:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530780:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696887.530789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704f800 x1874531145736448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696887.530790:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696887.530802:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696887.530833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145736448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.530850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.530856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.530887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696887.530896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704ea00 x1874531145736320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.530908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145736320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.530916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.530920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.530924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696887.530931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704d880 x1874531145736192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.530944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145736192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.530951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.530954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.530957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696887.530965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704df80 x1874531145736064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.530976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145736064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.530984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.530987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.530990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145736064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696887.530996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704ce00 x1874531145735936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.531020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145735936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.531028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.531031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.531035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696887.531040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704e680 x1874531145735808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.531050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145735808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.531056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.531059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.531062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696887.531068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704c000 x1874531145735552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.531078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145735552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.531085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.531087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.531091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696887.531096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704c380 x1874531145735680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.531113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145735680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.531120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.531123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.531126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145735680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696887.531133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696884/real 1787696887] req@ffff8f518704c700 x1874531145735424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696887.531144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145735424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696887.531150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696887.531153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696887.531156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145735424:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787696888.426516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696889.578482:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578496:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696889.578500:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578508:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578517:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578523:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696889.578556:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145736576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578579:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696889.578580:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578583:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578585:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578587:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578600:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696889.578608:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145736704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696889.578619:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696889.578623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145736576:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696889.578624:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578627:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578632:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578636:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696889.578643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145736832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696889.578652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578654:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578657:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578659:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578661:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696889.578668:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145736960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578673:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696889.578676:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578678:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578681:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578684:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578687:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696889.578693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787696889.578699:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145737088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578717:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696889.578719:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578727:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578732:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696889.578740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704c700 x1874531145736576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696889.578750:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145737216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696889.578765:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578769:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578773:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578777:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696889.578781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145736576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696889.578783:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696889.578798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787696889.578799:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145737344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696889.578808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787696889.578815:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696889.578818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145736576:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696889.578820:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696889.578845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578851:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578857:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578872:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578877:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696889.578898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145736704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696889.578899:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145737472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696889.578915:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696889.578921:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696889.578923:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696889.578927:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4789 00000100:00080000:2.0:1787696889.578931:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696889.578935:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696889.578940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696889.578942:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696889.578956:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145737600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696889.578963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704c380 x1874531145736704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.578979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145736704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.578987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.578990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.578994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145736704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696889.579001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145736832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696889.579020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704c000 x1874531145736832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145736832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145736832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696889.579061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145736960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696889.579077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704e680 x1874531145736960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145736960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145736960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696889.579115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145737088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696889.579132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704ce00 x1874531145737088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145737088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145737088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696889.579171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696889.579188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704df80 x1874531145737216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145737216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696889.579225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696889.579242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704d880 x1874531145737344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145737344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696889.579292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696889.579310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704ea00 x1874531145737472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145737472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696889.579348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696889.579403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696889.579415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696889/real 1787696889] req@ffff8f518704f800 x1874531145737600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696889.579425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145737600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696889.579432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696889.579435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696889.579438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145737600:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696890.474534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696891.498547:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696892.522529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696894.570543:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696894.698495:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698503:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696894.698507:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698515:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698525:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696894.698531:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696894.698564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145737728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696894.698588:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698598:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696894.698600:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698604:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698608:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696894.698612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.698622:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696894.698631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145737728:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696894.698636:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145737856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696894.698647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696894.698652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696894.698663:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.698667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696894.698678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145737984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696894.698688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696894.698690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696894.698692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696894.698695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698699:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698702:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696894.698706:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.698710:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696894.698720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145738112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696894.698728:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698732:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696894.698733:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698750:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698754:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696894.698758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f5187389c00 x1874531145737728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696894.698761:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.698769:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696894.698783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145738240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696894.698788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145737728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696894.698801:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698810:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696894.698814:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696894.698816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696894.698820:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698824:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696894.698828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696894.698832:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696894.698836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145737728:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787696894.698839:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696894.698853:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145738368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696894.698862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145737856:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696894.698865:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696894.698873:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698877:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698880:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696894.698884:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.698889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696894.698892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696894.698902:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145738496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696894.698910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f5187388380 x1874531145737856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696894.698915:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696894.698921:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696894.698925:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698931:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698934:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696894.698935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145737856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696894.698940:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.698948:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696894.698952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.698959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787696894.698960:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145738624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696894.698966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145737856:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696894.698978:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696894.698979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145737984:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696894.698983:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696894.698987:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696894.698992:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4794 00000100:00080000:3.0:1787696894.698995:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696894.698999:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696894.699004:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696894.699013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696894.699017:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145738752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696894.699028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f518738aa00 x1874531145737984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145737984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145737984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696894.699063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145738112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696894.699080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696894.699088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f518738b480 x1874531145738112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145738112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145738112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696894.699123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145738240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696894.699138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696894.699149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f518738ad80 x1874531145738240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145738240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145738240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696894.699195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696894.699211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696894.699221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f518738bb80 x1874531145738368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145738368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696894.699252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696894.699265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696894.699274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f518738b800 x1874531145738496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145738496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696894.699304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696894.699318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696894.699326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f5187388000 x1874531145738624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145738624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696894.699388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696894.699406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696894.699416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696894/real 1787696894] req@ffff8f5187389c00 x1874531145738752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696894.699427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145738752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696894.699435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696894.699438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696894.699441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145738752:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696895.594515:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696896.618499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696898.666525:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696899.690520:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696899.818496:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.818512:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696899.818517:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696899.818531:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.818544:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.818553:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696899.818603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145738880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696899.818643:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.818664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696899.818667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696899.818672:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.818677:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.818682:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696899.818693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696899.818710:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145739008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696899.818716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145738880:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696899.818730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.818735:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696899.818737:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696899.818742:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.818747:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.818751:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696899.818757:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696899.818773:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145739136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696899.818785:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.818790:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787696899.818794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696899.818795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696899.818802:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.818807:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.818812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696899.818818:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696899.818841:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145739264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696899.818852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f518738a300 x1874531145738880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696899.818862:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.818868:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696899.818871:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696899.818876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145738880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696899.818878:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.818886:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696899.818891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696899.818893:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696899.818897:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.818902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145738880:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787696899.818903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696899.818921:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145739392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696899.818935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739008:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696899.818935:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.818944:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696899.818948:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696899.818953:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.818959:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.818964:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696899.818970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696899.818971:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696899.818986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f5187388a80 x1874531145739008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696899.818991:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145739520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696899.819003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145739008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696899.819009:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696899.819013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696899.819015:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696899.819020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696899.819020:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696899.819025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739008:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696899.819030:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00100000:1.0:1787696899.819033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739136:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696899.819038:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.819045:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696899.819051:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696899.819052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f518738a680 x1874531145739136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696899.819071:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145739648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696899.819083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145739136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696899.819086:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696899.819097:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696899.819101:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696899.819104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696899.819109:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:1.0:1787696899.819113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696899.819116:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696899.819121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739136:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696899.819126:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739264:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787696899.819136:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696899.819154:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145739776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696899.819158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f5187388e00 x1874531145739264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696899.819192:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696899.819193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145739264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696899.819203:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787696899.819207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696899.819208:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696899.819211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.819220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739264:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787696899.819220:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4799 00000100:00080000:3.0:1787696899.819230:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696899.819237:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739392:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787696899.819244:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696899.819259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696899.819264:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145739904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696899.819270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f5187389180 x1874531145739392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696899.819292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145739392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696899.819303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696899.819305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.819308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145739392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696899.819313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696899.819324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f5187389500 x1874531145739520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696899.819347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145739520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696899.819394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696899.819398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.819401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696899.819407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696899.819428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f518738b100 x1874531145739648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696899.819452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145739648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696899.819460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696899.819463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.819480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696899.819486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696899.819506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f5187389f80 x1874531145739776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696899.819536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145739776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696899.819543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696899.819546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.819550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696899.819566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696899.819586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696899.819596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696899/real 1787696899] req@ffff8f5187388e00 x1874531145739904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696899.819605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145739904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696899.819611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696899.819613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696899.819615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145739904:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696900.714562:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696902.762522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696903.786531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696904.810550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696904.938642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145740032:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696904.938738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.938790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace91180 x1874531145740032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.938838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145740032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.938880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.938891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.938904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145740032:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696904.938929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696904.938973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.938994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace91c00 x1874531145740160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145740160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696904.939134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696904.939170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace91500 x1874531145740288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145740288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696904.939230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696904.939253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace92300 x1874531145740416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145740416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696904.939298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696904.939320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace92d80 x1874531145740544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145740544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145740544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696904.939422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145740672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696904.939446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace93480 x1874531145740672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145740672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145740672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696904.939494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145740800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696904.939517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace91880 x1874531145740800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145740800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145740800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696904.939556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145740928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696904.939594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace90700 x1874531145740928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145740928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145740928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696904.939642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145741056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696904.939660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696904.939669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696904/real 1787696904] req@ffff8f51ace90380 x1874531145741056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696904.939677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145741056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696904.939682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696904.939685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696904.939687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145741056:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696906.858492:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696907.882531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696908.906479:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696910.058596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145741184:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696910.058656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.058701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace90000 x1874531145741184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.058722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145741184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.058743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.058752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.058758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145741184:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696910.058771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696910.058809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.058831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace92680 x1874531145741312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.058851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145741312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.058864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.058872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.058880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696910.058911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696910.058940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.058949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace90e00 x1874531145741440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.058962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145741440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.058968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.058970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.058972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696910.058982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696910.058995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.059002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace93b80 x1874531145741568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.059010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145741568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.059015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.059017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.059019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696910.059022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696910.059032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.059037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace93100 x1874531145741696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.059044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145741696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.059054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.059056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.059059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145741696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696910.059065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145741824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696910.059083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.059095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace93800 x1874531145741824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.059109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145741824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.059118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.059122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.059125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145741824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696910.059129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145741952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696910.059147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.059159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace90380 x1874531145741952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.059172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145741952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.059181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.059184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.059189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145741952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696910.059195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145742080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696910.059219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.059227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace90700 x1874531145742080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.059235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145742080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.059240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.059242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.059245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145742080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696910.059248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145742208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696910.059258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696910.059263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696910/real 1787696910] req@ffff8f51ace91880 x1874531145742208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696910.059273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145742208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696910.059281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696910.059284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696910.059287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145742208:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696910.954593:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696911.978556:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696913.002600:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696915.050599:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696915.178509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.178519:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696915.178524:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178534:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.178546:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.178554:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696915.178604:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145742336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696915.178635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.178638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696915.178641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178646:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.178650:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.178655:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.178665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696915.178679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145742464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696915.178692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145742336:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696915.178693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.178702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696915.178705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178710:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.178713:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.178717:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.178722:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696915.178737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145742592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696915.178747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.178750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696915.178752:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178757:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.178761:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787696915.178767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696915.178767:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.178777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696915.178790:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145742720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696915.178800:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.178804:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696915.178806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178810:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.178814:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.178818:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.178823:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696915.178834:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145742848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696915.178844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f518738a680 x1874531145742336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696915.178846:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.178861:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696915.178863:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178870:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00100000:1.0:1787696915.178873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145742336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696915.178876:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.178885:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696915.178892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787696915.178893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696915.178902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787696915.178912:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145742976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696915.178913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145742336:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696915.178941:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696915.178943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696915.178949:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696915.178953:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.178959:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.178965:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.178969:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.178976:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696915.178980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696915.178994:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145743104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696915.179001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f5187388a80 x1874531145742464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696915.179009:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696915.179018:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696915.179022:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.179029:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00100000:1.0:1787696915.179034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145742464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696915.179036:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.179044:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.179053:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696915.179055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787696915.179067:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145743232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696915.179071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696915.179087:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696915.179088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696915.179094:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696915.179099:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696915.179105:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4814 00000100:00080000:3.0:1787696915.179109:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696915.179113:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696915.179118:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696915.179124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696915.179132:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145743360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696915.179143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f518738a300 x1874531145742592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145742592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696915.179183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696915.179204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696915.179216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f5187389c00 x1874531145742720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145742720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696915.179262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696915.179283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696915.179295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f5187388000 x1874531145742848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145742848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145742848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696915.179346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145742976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696915.179417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696915.179431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f518738b800 x1874531145742976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145742976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145742976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696915.179464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145743104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696915.179481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696915.179492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f518738bb80 x1874531145743104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145743104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145743104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696915.179525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145743232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696915.179541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696915.179551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f518738ad80 x1874531145743232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145743232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145743232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696915.179583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145743360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696915.179601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696915.179611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696915/real 1787696915] req@ffff8f518738b480 x1874531145743360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696915.179623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145743360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696915.179631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696915.179634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696915.179637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145743360:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696916.074561:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696917.098592:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696919.146526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696920.170529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696920.298531:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298543:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696920.298548:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298564:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298577:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298585:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696920.298630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145743488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298662:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696920.298664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298667:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298670:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298673:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696920.298691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145743616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298699:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696920.298703:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298706:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298710:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298714:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696920.298727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145743744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696920.298737:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298740:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298742:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298745:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696920.298762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145743488:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696920.298764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145743872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696920.298792:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298796:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298798:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298801:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298805:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696920.298812:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145744000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298818:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298821:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696920.298823:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298827:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298831:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298833:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298837:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696920.298845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145744128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696920.298855:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298858:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298861:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298869:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696920.298876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298877:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696920.298887:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145744256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298902:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298906:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696920.298908:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696920.298915:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298920:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298925:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.298933:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696920.298941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f518738aa00 x1874531145743488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696920.298952:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145744384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696920.298964:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696920.298967:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696920.298973:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696920.298974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145743488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696920.298980:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4819 00000100:00080000:3.0:1787696920.298987:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696920.298997:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696920.299009:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696920.299012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787696920.299021:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145744512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696920.299029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145743488:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696920.299048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145743616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696920.299087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f5187388380 x1874531145743616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145743616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145743616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696920.299143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145743744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696920.299170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f518738b480 x1874531145743744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145743744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145743744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696920.299231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145743872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696920.299255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f518738ad80 x1874531145743872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145743872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145743872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696920.299284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145744000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696920.299305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f518738bb80 x1874531145744000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145744000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145744000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696920.299394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696920.299411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f518738b800 x1874531145744128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145744128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696920.299449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696920.299460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f5187388000 x1874531145744256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145744256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696920.299488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696920.299498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f5187389c00 x1874531145744384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145744384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696920.299523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696920.299539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696920.299545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696920/real 1787696920] req@ffff8f518738a300 x1874531145744512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696920.299553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145744512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696920.299558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696920.299560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696920.299562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145744512:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696921.194490:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696923.242540:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696924.266567:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696925.290582:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696925.418474:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.418485:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696925.418490:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418500:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418511:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418518:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696925.418554:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145744640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696925.418582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.418587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696925.418589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418597:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418601:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696925.418611:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696925.418623:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145744768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696925.418634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.418637:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696925.418639:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418644:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418647:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696925.418654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145744640:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787696925.418659:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696925.418675:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145744896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696925.418685:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.418689:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696925.418691:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418695:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418699:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418703:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696925.418708:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696925.418718:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145745024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696925.418736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696925.418742:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.418747:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696925.418750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418756:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418761:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418770:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696925.418775:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696925.418787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f5187388a80 x1874531145744640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696925.418791:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145745152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696925.418821:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.418824:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696925.418826:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696925.418833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145744640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696925.418833:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418841:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418848:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696925.418857:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696925.418860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.418867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787696925.418872:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145745280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696925.418877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145744640:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696925.418891:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696925.418895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145744768:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696925.418897:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696925.418901:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418908:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.418912:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.418916:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696925.418922:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696925.418929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696925.418936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145745408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696925.418947:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696925.418948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f518738a680 x1874531145744768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696925.418956:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696925.418961:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.418970:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00100000:1.0:1787696925.418976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145744768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696925.418976:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696925.418989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696925.418992:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696925.418998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787696925.419000:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696925.419005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145744768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696925.419019:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145745536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696925.419030:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696925.419033:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696925.419038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145744896:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696925.419038:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696925.419048:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4825 00000100:00080000:3.0:1787696925.419060:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696925.419065:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696925.419071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696925.419072:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696925.419087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f5187388e00 x1874531145744896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696925.419089:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145745664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696925.419132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145744896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145744896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696925.419158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145745024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696925.419182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696925.419196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f5187389f80 x1874531145745024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696925.419227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145745024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145745024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696925.419251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145745152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696925.419273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696925.419286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f518738b100 x1874531145745152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696925.419299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145745152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145745152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696925.419328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696925.419351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696925.419396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f5187389500 x1874531145745280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696925.419409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145745280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696925.419432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696925.419454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696925.419466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f5187389180 x1874531145745408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696925.419479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145745408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696925.419511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696925.419534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696925.419547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f518738a300 x1874531145745536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696925.419561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145745536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696925.419599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696925.419620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696925.419631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696925/real 1787696925] req@ffff8f5187389c00 x1874531145745664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696925.419644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145745664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696925.419653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696925.419657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696925.419660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145745664:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696927.338541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696928.362492:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696929.386518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696930.538417:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538429:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696930.538435:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538447:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538460:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538468:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696930.538523:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145745792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696930.538561:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538564:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696930.538566:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538570:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538573:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538577:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538587:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696930.538602:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145745920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696930.538614:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696930.538620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538627:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00100000:1.0:1787696930.538628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145745792:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696930.538632:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538640:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538645:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696930.538661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145746048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696930.538668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696930.538673:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538676:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538679:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538682:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538685:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696930.538693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145746176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696930.538700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696930.538705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696930.538706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696930.538710:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538713:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538716:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538721:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538725:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696930.538739:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145746304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696930.538749:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538756:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787696930.538757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f5187388000 x1874531145745792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696930.538758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538768:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538776:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538784:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538790:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696930.538792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145745792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.538808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696930.538808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145746432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696930.538815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.538820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145745792:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696930.538837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538840:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696930.538842:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538846:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538849:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538852:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538857:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696930.538859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145745920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696930.538869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145746560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696930.538877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538881:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696930.538882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538885:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538888:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696930.538891:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696930.538897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787696930.538898:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696930.538918:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145746688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696930.538920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f518738b800 x1874531145745920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696930.538943:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696930.538947:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696930.538950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145745920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696930.538950:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696930.538961:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4830 00000100:00080000:3.0:1787696930.538968:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696930.538971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696930.538976:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696930.538981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787696930.538985:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696930.538988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145745920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696930.538999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145746048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696930.539004:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145746816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696930.539030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f518738bb80 x1874531145746048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145746048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145746048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696930.539075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145746176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696930.539097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f518738ad80 x1874531145746176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145746176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145746176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696930.539145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145746304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696930.539161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f518738b480 x1874531145746304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145746304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145746304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696930.539194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696930.539204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f5187388380 x1874531145746432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145746432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696930.539240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696930.539251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f518738aa00 x1874531145746560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145746560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696930.539279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696930.539289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f5187388000 x1874531145746688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145746688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696930.539313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696930.539322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696930.539328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696930/real 1787696930] req@ffff8f5187389c00 x1874531145746816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696930.539335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145746816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696930.539341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696930.539342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696930.539344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145746816:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696931.434567:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696932.458621:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696933.482584:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696935.530579:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696935.658658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145746944:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696935.658723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.658768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace93480 x1874531145746944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.658808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145746944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.658837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.658844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.658853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145746944:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696935.658882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696935.658910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.658924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace92d80 x1874531145747072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.658940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145747072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.658953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.658959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.658963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696935.658973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696935.658996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace92300 x1874531145747200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145747200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696935.659059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696935.659083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace91500 x1874531145747328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145747328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696935.659131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696935.659150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace91c00 x1874531145747456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145747456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145747456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696935.659185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696935.659196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace91180 x1874531145747584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145747584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696935.659237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696935.659257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace91880 x1874531145747712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145747712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696935.659329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696935.659352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace90700 x1874531145747840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145747840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696935.659412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696935.659427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696935.659433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696935/real 1787696935] req@ffff8f51ace90380 x1874531145747968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696935.659443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145747968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696935.659457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696935.659460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696935.659464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145747968:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696936.554542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696937.578541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696939.626485:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696940.650579:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696940.778467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.778478:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696940.778482:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696940.778492:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.778503:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.778511:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696940.778548:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145748096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696940.778579:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.778583:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696940.778586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696940.778591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.778594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.778598:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696940.778608:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696940.778623:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145748224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696940.778633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145748096:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696940.778637:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.778642:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696940.778645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696940.778650:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.778655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.778659:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696940.778665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696940.778680:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145748352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696940.778692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696940.778692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.778698:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696940.778701:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696940.778707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.778711:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.778716:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696940.778722:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696940.778732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f518738a300 x1874531145748096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696940.778738:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145748480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696940.778753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696940.778754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145748096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696940.778764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696940.778769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696940.778770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.778780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696940.778780:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00100000:1.0:1787696940.778787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145748096:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696940.778788:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.778798:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696940.778804:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696940.778818:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145748608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696940.778830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748224:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696940.778850:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.778855:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696940.778857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696940.778861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696940.778867:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.778872:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.778878:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696940.778880:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f5187389180 x1874531145748224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696940.778885:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696940.778895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145748224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.778903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696940.778905:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145748736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696940.778908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.778914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696940.778925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696940.778931:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.778936:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696940.778939:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696940.778958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696940.778963:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.778969:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696940.778971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f5187389500 x1874531145748352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696940.778977:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696940.778984:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696940.778990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145748352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.778998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696940.779002:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145748864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696940.779003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696940.779018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748480:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787696940.779027:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.779032:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696940.779034:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696940.779039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696940.779041:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.779047:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696940.779052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f518738b100 x1874531145748480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696940.779056:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696940.779069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145748480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696940.779069:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696940.779081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.779084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787696940.779092:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145748992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696940.779104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748608:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787696940.779106:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696940.779112:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696940.779114:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696940.779120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696940.779121:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4840 00000100:00080000:3.0:1787696940.779128:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696940.779133:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696940.779138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f5187389f80 x1874531145748608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696940.779139:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696940.779157:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145749120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696940.779161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145748608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.779188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.779191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145748608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696940.779198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145748736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696940.779215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696940.779223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f5187388e00 x1874531145748736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696940.779230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145748736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.779236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.779238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145748736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696940.779243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145748864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696940.779253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696940.779259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f518738a680 x1874531145748864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696940.779266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145748864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.779271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.779273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145748864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696940.779285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145748992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696940.779295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696940.779301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f5187388a80 x1874531145748992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696940.779308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145748992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.779314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.779317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145748992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696940.779326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145749120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696940.779344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696940.779381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696940/real 1787696940] req@ffff8f518738b100 x1874531145749120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696940.779391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145749120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696940.779397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696940.779399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696940.779401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145749120:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696941.674540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696943.722548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696944.746573:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696945.770573:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696945.898477:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696945.898491:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898502:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898513:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898519:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696945.898561:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145749248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696945.898590:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696945.898594:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898598:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898600:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898610:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696945.898618:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145749376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696945.898630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696945.898634:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898636:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898638:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898640:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696945.898650:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145749504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696945.898656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145749248:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787696945.898658:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898665:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696945.898667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898671:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898673:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696945.898684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145749632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696945.898690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696945.898694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898696:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898701:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696945.898717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145749760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696945.898723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898725:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696945.898726:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898729:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898731:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898734:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898736:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696945.898743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145749888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696945.898750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696945.898752:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898755:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696945.898757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898760:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898762:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898765:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696945.898774:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145750016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696945.898783:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696945.898786:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898789:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898790:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898792:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898795:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696945.898802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145750144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696945.898809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704d180 x1874531145749248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696945.898810:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696945.898815:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696945.898817:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696945.898824:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4845 00000100:00080000:2.0:1787696945.898828:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696945.898831:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696945.898834:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696945.898845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145749248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787696945.898845:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145750272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696945.898875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.898884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.898889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145749248:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696945.898897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696945.898922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.898935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704ed80 x1874531145749376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.898947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145749376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.898955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.898969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.898976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696945.899003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696945.899022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704dc00 x1874531145749504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145749504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696945.899070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696945.899088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704fb80 x1874531145749632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145749632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696945.899127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696945.899140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704d500 x1874531145749760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145749760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145749760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696945.899172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145749888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696945.899183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704f480 x1874531145749888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145749888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145749888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696945.899207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145750016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696945.899216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704e300 x1874531145750016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145750016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145750016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696945.899249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145750144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696945.899260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704f800 x1874531145750144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145750144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145750144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696945.899282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145750272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696945.899291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696945.899296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696945/real 1787696945] req@ffff8f518704ea00 x1874531145750272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696945.899302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145750272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696945.899307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696945.899308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696945.899310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145750272:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696947.818528:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696948.842580:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696949.866518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696951.018524:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018535:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696951.018539:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018549:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018558:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018565:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696951.018609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145750400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696951.018651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018662:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.018672:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696951.018683:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145750528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018693:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018696:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696951.018698:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018703:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018706:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018710:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.018715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696951.018719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145750400:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696951.018730:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145750656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018741:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018744:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696951.018746:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018749:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018753:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018757:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.018761:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696951.018797:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145750784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018806:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696951.018811:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018815:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018819:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018822:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.018827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696951.018836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145750912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018845:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696951.018850:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018854:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018857:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018861:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.018866:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696951.018867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696951.018881:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145751040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018891:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018894:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696951.018896:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018900:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018903:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018907:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696951.018912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750656:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787696951.018912:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696951.018928:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145751168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018939:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696951.018944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750784:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787696951.018945:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696951.018949:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696951.018955:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.018959:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.018963:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.018968:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696951.018973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787696951.018981:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145751296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696951.018991:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696951.018994:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696951.018996:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696951.018999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751040:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696951.019002:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4850 00000100:00080000:3.0:1787696951.019009:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696951.019013:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696951.019018:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696951.019030:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145751424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696951.019038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696951.019063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696951.019082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751424:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696951.914606:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696952.938550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696953.962568:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787696954.090849:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787696954.090908:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696954.090930:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.090992:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091008:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091021:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091029:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091037:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091045:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091056:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787696954.091064:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787696954.091160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f518738ad80 x1874531145751424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145751424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.091247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f5187388380 x1874531145751168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145751168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696954.091292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f518738b480 x1874531145751296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145751296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.091331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f518738a300 x1874531145750656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145750656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696954.091388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f5187389c00 x1874531145750784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145750784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.091421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f5187388000 x1874531145750912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145750912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696954.091455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f518738aa00 x1874531145751040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145751040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145751040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696954.091496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f5187389500 x1874531145750400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145750400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145750400:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696954.091530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696951/real 1787696954] req@ffff8f5187389180 x1874531145750528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696954.091540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145750528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696954.091547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696954.091550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696954.091554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145750528:192.168.201.125@tcp:38:kworker.0 00010000:00010000:1.0:1787696956.010591:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696956.138448:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138455:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696956.138458:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696956.138468:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138478:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138485:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696956.138526:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145751552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138557:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696956.138559:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696956.138564:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138572:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696956.138579:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696956.138590:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145751680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138597:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138600:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696956.138601:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696956.138604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145751552:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696956.138606:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138610:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138615:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696956.138618:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696956.138625:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145751808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138631:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138633:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696956.138634:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696956.138636:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138638:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138640:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696956.138642:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696956.138648:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145751936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138654:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696956.138657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696956.138659:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138661:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138663:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696956.138665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696956.138670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696956.138672:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145752064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138696:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696956.138698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696956.138703:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138707:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138711:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696956.138716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696956.138722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f518738bb80 x1874531145751552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696956.138730:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145752192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138739:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138742:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696956.138744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696956.138746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145751552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696956.138750:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138756:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138763:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696956.138765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787696956.138770:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696956.138772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.138779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145751552:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696956.138786:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145752320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138799:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696956.138801:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696956.138804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145751680:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696956.138808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138815:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138819:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696956.138825:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696956.138843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696956.138846:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145752448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696956.138858:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696956.138862:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696956.138865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696956.138866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f518738b800 x1874531145751680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696956.138873:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4855 00000100:00080000:3.0:1787696956.138879:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696956.138886:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696956.138892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145751680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696956.138893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696956.138905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696956.138909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145752576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696956.138912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.138921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145751680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696956.138931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145751808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696956.138951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.138963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f5187389180 x1874531145751808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.138976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145751808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.138985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.138988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.138991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145751808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696956.138997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145751936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696956.139016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.139027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f5187389500 x1874531145751936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.139038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145751936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.139045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.139048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.139051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145751936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696956.139057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145752064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696956.139074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.139084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f518738aa00 x1874531145752064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.139096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145752064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.139103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.139106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.139109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145752064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696956.139114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696956.139131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.139140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f5187388000 x1874531145752192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.139150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145752192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.139158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.139161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.139164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696956.139169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696956.139184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.139193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f5187389c00 x1874531145752320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.139203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145752320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.139211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.139214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.139217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696956.139236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696956.139253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.139263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f518738a300 x1874531145752448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.139274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145752448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.139281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.139284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.139287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696956.139293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696956.139309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696956.139318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696956/real 1787696956] req@ffff8f518738b480 x1874531145752576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696956.139330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145752576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696956.139337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696956.139340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696956.139343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145752576:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696957.034535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696958.058594:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696960.106510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696961.130514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787696961.258636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.258645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787696961.258649:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.258659:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00080000:2.0:1787696961.258671:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.258678:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787696961.258712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145752704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696961.258739:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.258752:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696961.258754:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.258759:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00080000:2.0:1787696961.258762:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.258767:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696961.258776:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696961.258791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145752704:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696961.258791:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145752832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696961.258807:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.258809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696961.258811:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.258816:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00080000:2.0:1787696961.258819:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.258822:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696961.258827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696961.258839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145752960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696961.258847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.258852:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696961.258853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.258860:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4860 00000800:00000100:1.0:1787696961.258861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696961.258864:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.258869:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696961.258875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696961.258887:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145753088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696961.258895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704d880 x1874531145752704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696961.258897:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.258905:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696961.258908:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.258914:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00100000:1.0:1787696961.258919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145752704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696961.258919:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.258927:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787696961.258931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787696961.258933:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696961.258939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.258944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145752704:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787696961.258947:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145753216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696961.258962:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.258965:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787696961.258967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.258971:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00080000:2.0:1787696961.258975:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696961.258978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145752832:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696961.258982:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696961.258989:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787696961.259003:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145753344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696961.259004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787696961.259017:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696961.259019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704df80 x1874531145752832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696961.259023:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787696961.259026:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.259034:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00080000:2.0:1787696961.259039:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.259045:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145752832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787696961.259051:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787696961.259063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787696961.259064:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145753472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696961.259071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145752832:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696961.259080:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787696961.259088:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787696961.259090:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696961.259092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145752960:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787696961.259103:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00080000:2.0:1787696961.259108:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787696961.259113:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787696961.259117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787696961.259119:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787696961.259131:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145753600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787696961.259142:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696961.259143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704ce00 x1874531145752960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696961.259148:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787696961.259152:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787696961.259159:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4860 00000100:00100000:1.0:1787696961.259163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145752960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787696961.259164:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787696961.259173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787696961.259173:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787696961.259178:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696961.259181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145752960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787696961.259192:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145753728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696961.259195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145753088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696961.259219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704e680 x1874531145753088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696961.259237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145753088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696961.259241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696961.259243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145753088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696961.259248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145753216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696961.259261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704c000 x1874531145753216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696961.259280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145753216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696961.259287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696961.259290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145753216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696961.259308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696961.259321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704c380 x1874531145753344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696961.259334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145753344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696961.259343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696961.259345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696961.259350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696961.259377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704c700 x1874531145753472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696961.259393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145753472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696961.259400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696961.259403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696961.259412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696961.259429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704d880 x1874531145753600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696961.259445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145753600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696961.259449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696961.259451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696961.259456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696961.259466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696961.259471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696961/real 1787696961] req@ffff8f518704ea00 x1874531145753728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696961.259478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145753728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696961.259482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696961.259483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696961.259485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145753728:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696962.154546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696964.202525:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696965.226549:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696966.250474:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696966.378455:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696966.378463:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696966.378467:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696966.378476:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378485:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378491:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696966.378545:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145753856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696966.378568:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696966.378572:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696966.378574:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696966.378578:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378581:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378585:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696966.378594:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696966.378608:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145753984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696966.378609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145753856:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696966.378625:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696966.378628:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696966.378630:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696966.378633:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378639:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696966.378643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696966.378655:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145754112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696966.378663:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696966.378666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696966.378667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696966.378671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378674:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378677:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696966.378682:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696966.378684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696966.378694:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145754240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696966.378705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696966.378708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696966.378709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696966.378713:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378716:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378719:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696966.378724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f5187388380 x1874531145753856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696966.378725:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696966.378739:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145754368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696966.378745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145753856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696966.378760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696966.378763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696966.378764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696966.378768:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696966.378770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696966.378774:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00100000:1.0:1787696966.378776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145753856:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696966.378778:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378784:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696966.378788:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696966.378798:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145754496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696966.378799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145753984:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696966.378813:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696966.378817:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696966.378819:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696966.378824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696966.378825:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378831:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696966.378835:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787696966.378838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f518738ad80 x1874531145753984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787696966.378840:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696966.378855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145754624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696966.378858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145753984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696966.378876:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787696966.378879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696966.378880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696966.378883:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787696966.378888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696966.378890:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00100000:1.0:1787696966.378895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145753984:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696966.378900:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696966.378903:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145754112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696966.378908:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696966.378914:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696966.378926:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145754752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696966.378930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696966.378940:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696966.378942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f518738b100 x1874531145754112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696966.378946:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696966.378950:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696966.378957:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4866 00000100:00080000:3.0:1787696966.378961:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696966.378965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145754112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696966.378967:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696966.378973:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787696966.378989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787696966.378993:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145754880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696966.378997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145754112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696966.379016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145754240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696966.379039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696966.379051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f5187388a80 x1874531145754240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696966.379067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145754240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696966.379072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696966.379074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145754240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696966.379080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145754368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696966.379092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696966.379099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f518738a680 x1874531145754368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696966.379105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145754368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696966.379117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696966.379119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145754368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696966.379133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696966.379145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696966.379155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f5187388e00 x1874531145754496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696966.379165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145754496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696966.379178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696966.379182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696966.379190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696966.379208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696966.379219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f5187389f80 x1874531145754624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696966.379226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145754624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696966.379231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696966.379233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696966.379238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696966.379249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696966.379254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f5187388380 x1874531145754752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696966.379261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145754752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696966.379265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696966.379266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696966.379271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696966.379279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696966.379284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696966/real 1787696966] req@ffff8f518738b480 x1874531145754880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696966.379290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145754880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696966.379294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696966.379296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696966.379306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145754880:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696968.298554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696969.322530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696970.346566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696971.498434:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498444:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696971.498448:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498458:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498467:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498474:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696971.498504:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145755008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696971.498533:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498535:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696971.498536:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498539:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498542:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498544:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498551:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696971.498558:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145755136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696971.498565:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498567:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696971.498568:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498571:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498574:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498576:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498579:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696971.498589:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145755264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696971.498590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145755008:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696971.498602:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498605:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696971.498606:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498608:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498609:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498611:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498614:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696971.498621:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145755392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696971.498627:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498629:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696971.498630:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498632:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498635:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498638:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498640:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696971.498646:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145755520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696971.498656:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787696971.498660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696971.498661:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696971.498664:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498670:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498674:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498677:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498680:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696971.498689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145755648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696971.498697:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498700:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696971.498705:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696971.498706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f518738a300 x1874531145755008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696971.498711:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498718:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498724:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498730:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696971.498734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145755008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696971.498745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145755776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696971.498748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.498758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696971.498761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787696971.498766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145755008:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696971.498767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696971.498772:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498778:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00100000:1.0:1787696971.498779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755136:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696971.498783:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696971.498790:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498795:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696971.498808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145755904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696971.498812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696971.498823:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696971.498826:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696971.498829:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696971.498834:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4871 00000100:00080000:3.0:1787696971.498838:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787696971.498839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f5187389c00 x1874531145755136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696971.498844:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696971.498852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696971.498860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145755136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696971.498868:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145756032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696971.498871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.498879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.498887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696971.498910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696971.498945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.498956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f5187388000 x1874531145755264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.498964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145755264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.498969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.498971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.498973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696971.498976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696971.498993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.499003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f518738aa00 x1874531145755392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.499020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145755392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.499031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.499033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.499036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696971.499043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696971.499068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.499078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f5187389500 x1874531145755520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.499088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145755520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.499095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.499098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.499101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145755520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696971.499106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145755648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696971.499124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.499135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f5187389180 x1874531145755648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.499144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145755648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.499156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.499159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.499163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145755648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696971.499168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145755776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696971.499183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.499195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f518738b800 x1874531145755776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.499201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145755776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.499206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.499214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.499217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145755776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696971.499222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145755904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696971.499239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.499249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f518738bb80 x1874531145755904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.499261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145755904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.499267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.499269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.499271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145755904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696971.499275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145756032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696971.499294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696971.499301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696971/real 1787696971] req@ffff8f518738b480 x1874531145756032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696971.499307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145756032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696971.499312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696971.499314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696971.499315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145756032:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696972.394625:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696973.418538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696974.442518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696976.490518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696976.618646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145756160:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696976.618700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.618745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace93800 x1874531145756160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.618771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145756160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.618781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.618785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.618788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145756160:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696976.618816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696976.618838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.618850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace93100 x1874531145756288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.618862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145756288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.618870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.618876:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.618881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696976.618889:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696976.618902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.618912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace93b80 x1874531145756416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.618939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145756416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.618947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.618953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.618958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696976.618967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696976.618983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.618993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace90e00 x1874531145756544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.619001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145756544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.619013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.619017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.619022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696976.619026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696976.619039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.619048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace92680 x1874531145756672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.619062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145756672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.619072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.619077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.619084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145756672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696976.619098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145756800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696976.619113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.619123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace90000 x1874531145756800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.619131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145756800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.619141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.619143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.619145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145756800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696976.619149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145756928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696976.619158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.619163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace93800 x1874531145756928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.619169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145756928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.619173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.619175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.619176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145756928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696976.619186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145757056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696976.619196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.619202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace90380 x1874531145757056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.619208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145757056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.619213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.619214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.619216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145757056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696976.619224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145757184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696976.619232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696976.619237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696976/real 1787696976] req@ffff8f51ace90700 x1874531145757184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696976.619243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145757184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696976.619247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696976.619249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696976.619250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145757184:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696977.514518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696978.538475:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696980.586490:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696981.610546:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696981.738456:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738463:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696981.738467:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696981.738474:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738482:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738487:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696981.738516:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145757312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696981.738541:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738543:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696981.738544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696981.738547:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738549:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738551:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738558:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696981.738566:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145757440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696981.738574:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738576:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696981.738578:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696981.738580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145757312:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696981.738582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738588:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738591:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738593:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696981.738602:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145757568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696981.738608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696981.738611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696981.738613:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738616:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738618:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696981.738628:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145757696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787696981.738635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696981.738636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696981.738641:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696981.738645:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738647:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738650:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738653:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696981.738660:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145757824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696981.738665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696981.738669:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696981.738673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f5187388380 x1874531145757312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696981.738674:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738680:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738687:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738694:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696981.738703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145757312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696981.738709:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145757952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787696981.738722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696981.738730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738734:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787696981.738737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696981.738739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696981.738746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145757312:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787696981.738747:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738755:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738761:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738766:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787696981.738774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787696981.738787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145758080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696981.738795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738797:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696981.738799:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787696981.738804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787696981.738804:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738812:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738815:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738820:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696981.738823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f5187389f80 x1874531145757440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696981.738835:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145758208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696981.738836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145757440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.738859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.738864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787696981.738864:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696981.738871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787696981.738872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757440:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696981.738876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787696981.738880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787696981.738881:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4881 00000100:00080000:3.0:1787696981.738888:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696981.738892:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696981.738896:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787696981.738902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.738912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f5187388e00 x1874531145757568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696981.738912:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145758336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696981.738936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145757568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.738940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.738945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.738949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696981.738954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696981.738966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.738972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f518738a680 x1874531145757696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696981.738979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145757696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.738983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.738985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.738986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696981.738990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696981.738998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.739003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f5187388a80 x1874531145757824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696981.739009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145757824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.739014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.739015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.739017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145757824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696981.739019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145757952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696981.739028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.739032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f518738b100 x1874531145757952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696981.739038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145757952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.739047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.739049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.739050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145757952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696981.739061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145758080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696981.739070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.739075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f518738ad80 x1874531145758080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696981.739081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145758080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.739085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.739087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.739088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145758080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696981.739092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145758208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696981.739100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.739104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f5187388380 x1874531145758208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696981.739110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145758208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.739114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.739116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.739118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145758208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696981.739121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145758336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696981.739129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696981.739134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696981/real 1787696981] req@ffff8f518738b480 x1874531145758336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696981.739140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145758336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696981.739144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696981.739146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696981.739147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145758336:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787696982.634537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696984.682540:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696985.706601:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696986.730542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696986.858446:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696986.858457:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858464:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858472:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858477:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696986.858504:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145758464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858527:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858530:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696986.858531:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858534:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858536:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858538:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858545:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696986.858552:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145758592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858562:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696986.858569:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858575:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858577:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858582:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858586:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696986.858594:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145758720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858600:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858602:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696986.858603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858606:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858608:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858614:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858619:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787696986.858620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145758464:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787696986.858633:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145758848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858642:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696986.858643:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858644:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858647:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858650:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858652:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696986.858660:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145758976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858664:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858667:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696986.858668:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858671:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858673:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858675:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858679:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696986.858686:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145759104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696986.858693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858695:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858698:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858701:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858705:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787696986.858718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787696986.858718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145759232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858732:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696986.858737:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858740:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858742:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858745:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696986.858753:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145759360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696986.858758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696986.858760:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696986.858760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696986.858763:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4886 00000100:00080000:3.0:1787696986.858765:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696986.858767:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696986.858769:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696986.858776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145759488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696986.858795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f518738bb80 x1874531145758464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.858820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145758464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.858830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.858833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.858836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145758464:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696986.858842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696986.858863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.858872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f518738b800 x1874531145758592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.858879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145758592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.858885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.858887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.858889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696986.858911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696986.858921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.858927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f5187389180 x1874531145758720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.858934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145758720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.858939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.858940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.858943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696986.858946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696986.858956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.858961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f5187389500 x1874531145758848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.858972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145758848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.858977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.858979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.858980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696986.858984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696986.858992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.858998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f518738aa00 x1874531145758976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.859008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145758976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.859015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.859018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.859020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145758976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696986.859026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696986.859046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.859057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f5187388000 x1874531145759104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.859064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145759104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.859074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.859076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.859078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696986.859082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696986.859094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.859100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f5187389c00 x1874531145759232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.859107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145759232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.859119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.859124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.859127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696986.859133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696986.859183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.859195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f518738a300 x1874531145759360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.859211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145759360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.859221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.859225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.859229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696986.859235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696986.859255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696986.859264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696986/real 1787696986] req@ffff8f518738b480 x1874531145759488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696986.859272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145759488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696986.859277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696986.859279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696986.859281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145759488:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696988.778619:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696989.802570:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696990.827315:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787696991.978473:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978484:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696991.978489:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978498:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978509:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978517:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787696991.978556:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145759616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978592:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978596:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696991.978598:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978603:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978606:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978611:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978620:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696991.978632:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145759744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696991.978645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978648:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978651:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978654:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696991.978670:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145759872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696991.978683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978686:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978689:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978692:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696991.978706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145760000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696991.978718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978721:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978724:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978727:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978732:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696991.978742:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145760128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978749:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978753:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696991.978754:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978772:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978776:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978780:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978785:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696991.978796:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145760256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978805:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978808:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696991.978809:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978813:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978816:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978818:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787696991.978831:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145760384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978839:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978842:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696991.978844:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978847:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978849:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978852:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978856:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696991.978865:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145760512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787696991.978874:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696991.978877:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696991.978879:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696991.978883:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4891 00000100:00080000:3.0:1787696991.978886:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787696991.978889:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787696991.978893:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787696991.978901:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145760640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787696991.979023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145759616:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696991.979117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f5187388380 x1874531145759616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145759616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145759616:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696991.979239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145759744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696991.979263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f518738ad80 x1874531145759744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145759744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145759744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696991.979309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145759872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696991.979328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f518738b100 x1874531145759872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145759872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145759872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696991.979401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145760000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696991.979422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f5187388a80 x1874531145760000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145760000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145760000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696991.979470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145760128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696991.979489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f518738a680 x1874531145760128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145760128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145760128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696991.979534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696991.979553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f5187388e00 x1874531145760256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145760256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696991.979602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696991.979622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f5187389f80 x1874531145760384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145760384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696991.979661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696991.979680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f518738b480 x1874531145760512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145760512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696991.979720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696991.979739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696991.979751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696991/real 1787696991] req@ffff8f518738a300 x1874531145760640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696991.979763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145760640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696991.979770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696991.979773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696991.979776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145760640:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696992.875404:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696993.898547:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696994.922732:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787696996.970534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787696997.100031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145760768:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787696997.100088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace91880 x1874531145760768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145760768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145760768:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787696997.100166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145760896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696997.100185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace91180 x1874531145760896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145760896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145760896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696997.100226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145761024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696997.100244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace91c00 x1874531145761024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145761024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145761024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696997.100283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145761152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696997.100299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace91500 x1874531145761152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145761152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145761152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696997.100339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145761280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696997.100381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace92300 x1874531145761280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145761280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145761280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696997.100718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696997.100740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace92d80 x1874531145761408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145761408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696997.100788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787696997.100805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace93480 x1874531145761536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145761536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787696997.100844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696997.100862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace92300 x1874531145761664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145761664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787696997.100900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787696997.100917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787696997.100927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696997/real 1787696997] req@ffff8f51ace91500 x1874531145761792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787696997.100939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145761792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787696997.100945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787696997.100948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787696997.100951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145761792:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787696997.994531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696999.018567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697001.066616:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697002.090550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697002.218456:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218463:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697002.218467:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218475:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218483:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218488:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697002.218525:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145761920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218545:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218547:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697002.218548:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218551:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218553:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218556:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218563:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697002.218570:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145762048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218577:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218579:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697002.218580:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218582:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218584:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218586:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218589:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697002.218597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145761920:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697002.218598:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145762176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218612:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697002.218613:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218615:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218617:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218618:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218621:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697002.218628:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145762304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218633:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218635:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697002.218636:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218639:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218642:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218644:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697002.218653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145762432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218660:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697002.218661:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697002.218665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697002.218667:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218671:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218675:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697002.218690:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145762560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697002.218704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218707:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00080000:3.0:1787697002.218711:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218715:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697002.218720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f5187389c00 x1874531145761920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697002.218721:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697002.218738:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145762688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697002.218744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145761920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697002.218752:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697002.218761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697002.218765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697002.218767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697002.218771:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4901 00000100:00100000:1.0:1787697002.218775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145761920:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697002.218776:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218782:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218790:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697002.218791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697002.218802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145762816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697002.218809:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697002.218812:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697002.218813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697002.218816:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4901 00000800:00000100:1.0:1787697002.218821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697002.218821:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697002.218826:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697002.218830:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697002.218836:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145762944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697002.218841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f5187388000 x1874531145762048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.218856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145762048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.218864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.218868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.218871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697002.218904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697002.218924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.218934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f518738aa00 x1874531145762176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.218945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145762176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.218953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.218956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.218958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697002.218963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697002.218981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.218991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f5187389500 x1874531145762304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.219004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145762304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.219011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.219014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.219017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697002.219022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697002.219037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.219046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f5187389180 x1874531145762432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.219058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145762432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.219065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.219068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.219071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145762432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697002.219076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697002.219092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.219101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f518738b800 x1874531145762560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.219111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145762560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.219125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.219128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.219131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697002.219136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697002.219153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.219163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f518738bb80 x1874531145762688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.219175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145762688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.219182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.219185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.219188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697002.219193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697002.219209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.219218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f518738a300 x1874531145762816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.219230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145762816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.219237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.219240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.219244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697002.219248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697002.219263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697002.219272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697002/real 1787697002] req@ffff8f518738b480 x1874531145762944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697002.219283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145762944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697002.219290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697002.219293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697002.219297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145762944:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697003.114589:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697005.162546:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697006.186569:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697007.210506:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697007.338420:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338430:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697007.338434:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338443:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00080000:3.0:1787697007.338453:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338468:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697007.338505:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145763072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697007.338527:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338530:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697007.338532:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338536:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00080000:3.0:1787697007.338539:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338544:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697007.338552:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697007.338558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145763072:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697007.338566:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145763200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697007.338576:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338579:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697007.338581:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338584:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00080000:3.0:1787697007.338587:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338590:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697007.338594:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697007.338605:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145763328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697007.338608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697007.338622:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338625:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697007.338627:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338631:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00080000:3.0:1787697007.338634:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338637:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697007.338642:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697007.338648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f5187389f80 x1874531145763072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697007.338654:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145763456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697007.338665:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697007.338669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145763072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697007.338670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697007.338674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338682:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00080000:1.0:1787697007.338684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697007.338686:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697007.338689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697007.338693:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697007.338694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145763072:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697007.338701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697007.338714:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145763584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697007.338716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763200:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697007.338731:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338735:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787697007.338737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697007.338739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338746:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00100000:1.0:1787697007.338749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f5187388e00 x1874531145763200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697007.338752:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338761:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697007.338765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145763200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697007.338766:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697007.338777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.338780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697007.338785:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145763712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697007.338788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697007.338807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697007.338813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338817:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697007.338819:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338824:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4906 00000800:00000100:1.0:1787697007.338828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697007.338828:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338835:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697007.338839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f518738a680 x1874531145763328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697007.338840:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697007.338852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145763328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697007.338857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145763840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697007.338861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.338866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.338872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697007.338877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338880:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697007.338883:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338889:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00100000:1.0:1787697007.338891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763456:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697007.338893:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338901:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697007.338906:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697007.338918:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145763968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697007.338921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697007.338936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f5187388a80 x1874531145763456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697007.338939:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697007.338946:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697007.338959:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697007.338963:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4906 00000100:00080000:3.0:1787697007.338969:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697007.338973:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697007.338978:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697007.338981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145763456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697007.338990:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145764096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697007.338993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.339002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.339009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697007.339029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697007.339048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697007.339059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f518738b100 x1874531145763584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697007.339071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145763584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697007.339078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.339081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.339084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145763584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697007.339090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145763712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697007.339107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697007.339117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f518738ad80 x1874531145763712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697007.339128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145763712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697007.339141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.339145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.339148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145763712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697007.339153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145763840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697007.339169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697007.339179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f5187388380 x1874531145763840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697007.339190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145763840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697007.339197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.339200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.339203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145763840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697007.339218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145763968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697007.339237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697007.339247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f5187389f80 x1874531145763968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697007.339258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145763968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697007.339265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.339268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.339271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145763968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697007.339277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145764096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697007.339294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697007.339312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697007/real 1787697007] req@ffff8f518738b480 x1874531145764096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697007.339323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145764096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697007.339329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697007.339332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697007.339335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145764096:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697009.258508:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697010.282558:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697011.306536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697012.458459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458466:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697012.458469:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458478:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458489:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458495:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697012.458534:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145764224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458551:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458554:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697012.458555:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458558:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458564:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458567:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458574:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697012.458582:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145764352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458588:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458590:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697012.458592:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458595:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458596:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458598:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458602:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697012.458609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145764480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697012.458618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458620:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458622:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458624:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458627:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697012.458635:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145764608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697012.458636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145764224:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697012.458646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697012.458652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458656:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458658:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458659:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458661:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697012.458667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145764736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458672:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458674:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697012.458675:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458677:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458679:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458682:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697012.458692:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145764864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458700:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458713:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697012.458715:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458719:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458724:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458728:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458733:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697012.458749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697012.458749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145764992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458765:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697012.458769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458773:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458776:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458780:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458784:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697012.458790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145765120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697012.458796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697012.458798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697012.458799:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697012.458802:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4912 00000100:00080000:3.0:1787697012.458804:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697012.458806:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697012.458810:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697012.458819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145765248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697012.458825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f518738a300 x1874531145764224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.458854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145764224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.458866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.458871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.458876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145764224:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697012.458885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697012.458912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.458926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f518738bb80 x1874531145764352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.458940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145764352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.458948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.458952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.458955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697012.458962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697012.458984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.458995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f518738b800 x1874531145764480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145764480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697012.459052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697012.459074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.459086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f5187389180 x1874531145764608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145764608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697012.459127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697012.459148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.459160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f5187389500 x1874531145764736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145764736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145764736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697012.459193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145764864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697012.459214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.459226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f518738aa00 x1874531145764864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145764864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145764864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697012.459266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145764992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697012.459285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.459297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f5187388000 x1874531145764992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145764992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145764992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697012.459330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145765120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697012.459350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.459398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f5187389c00 x1874531145765120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145765120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145765120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697012.459430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145765248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697012.459450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697012.459461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697012/real 1787697012] req@ffff8f518738b480 x1874531145765248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697012.459472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145765248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697012.459481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697012.459484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697012.459487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145765248:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697013.354521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697014.378523:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697015.402515:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697017.450484:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697017.578512:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.578528:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697017.578533:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.578548:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.578562:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.578572:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697017.578637:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145765376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.578672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.578676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697017.578679:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.578685:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.578695:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.578704:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.578716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697017.578731:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145765504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.578745:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.578754:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787697017.578755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145765376:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697017.578758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.578773:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.578778:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.578784:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.578790:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697017.578808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145765632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.578820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.578825:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697017.578827:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.578832:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.578837:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.578843:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.578849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697017.578865:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145765760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.578879:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697017.578880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765504:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697017.578897:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697017.578900:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.578909:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.578914:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.578921:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697017.578925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765632:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697017.578930:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697017.578955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697017.578957:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145765888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.578980:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.578985:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787697017.578989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765888:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697017.578989:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.579006:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.579012:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.579018:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.579024:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697017.579042:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145766016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.579060:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.579064:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697017.579067:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.579072:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00100000:1.0:1787697017.579078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766016:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697017.579080:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.579092:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.579098:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697017.579114:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145766144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.579139:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.579144:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697017.579146:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697017.579150:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.579155:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.579160:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.579168:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697017.579170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697017.579189:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145766272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697017.579212:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697017.579216:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697017.579220:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697017.579222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766272:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697017.579230:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4917 00000100:00080000:3.0:1787697017.579239:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697017.579245:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697017.579252:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697017.579269:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145766400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697017.579318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766400:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697018.474613:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697019.498619:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697020.650956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f5187389c00 x1874531145766400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145766400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697020.651075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f518738b480 x1874531145766272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145766272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697020.651120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f5187388e00 x1874531145766144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145766144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697020.651164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f518738a680 x1874531145766016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145766016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145766016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697020.651201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f5187388a80 x1874531145765888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145765888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697020.651236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f518738b100 x1874531145765760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145765760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697020.651279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f5187388380 x1874531145765504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145765504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697020.651311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f518738ad80 x1874531145765632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145765632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145765632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697020.651344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697017/real 1787697020] req@ffff8f5187389f80 x1874531145765376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697020.651389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145765376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697020.651399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697020.651402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697020.651405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145765376:192.168.201.125@tcp:250:kworker.0 00010000:00010000:1.0:1787697021.546528:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697022.570515:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697022.698639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145766528:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787697022.698735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.698793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace91c00 x1874531145766528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.698819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145766528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.698843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.698852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.698867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145766528:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697022.698902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145766656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697022.698942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.698978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace91180 x1874531145766656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145766656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145766656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697022.699052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145766784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697022.699091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace91880 x1874531145766784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145766784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145766784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697022.699144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145766912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697022.699163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace90700 x1874531145766912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145766912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145766912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697022.699205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145767040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697022.699223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace90380 x1874531145767040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145767040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145767040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697022.699264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697022.699283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace93800 x1874531145767168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145767168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697022.699349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697022.699396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace90000 x1874531145767296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145767296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697022.699445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697022.699463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace92680 x1874531145767424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145767424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697022.699507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697022.699541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697022.699554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697022/real 1787697022] req@ffff8f51ace90e00 x1874531145767552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697022.699566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145767552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697022.699582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697022.699585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697022.699588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145767552:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697023.594487:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697025.642562:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697026.666534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697027.690612:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697027.818544:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818562:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697027.818567:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818580:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.818594:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697027.818602:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697027.818651:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145767680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697027.818688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697027.818694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818701:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.818705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697027.818710:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.818721:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697027.818737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145767808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697027.818741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145767680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697027.818758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818761:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697027.818764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818769:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.818772:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697027.818776:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.818782:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697027.818796:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145767936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697027.818806:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818809:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697027.818811:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818815:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.818821:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697027.818829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697027.818829:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.818839:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697027.818851:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145768064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697027.818861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818865:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697027.818867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818872:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.818875:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697027.818879:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.818885:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697027.818889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f5187388000 x1874531145767680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697027.818900:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145768192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697027.818912:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818917:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697027.818918:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818923:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00100000:1.0:1787697027.818929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145767680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697027.818929:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697027.818937:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.818946:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697027.818954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.818961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697027.818961:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145768320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697027.818966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145767680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697027.818982:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697027.818986:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697027.818988:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.818992:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.818997:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697027.819002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145767808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697027.819005:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.819012:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697027.819025:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145768448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697027.819036:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697027.819041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697027.819043:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697027.819046:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.819053:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.819057:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697027.819060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f518738aa00 x1874531145767808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697027.819064:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697027.819072:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697027.819089:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145768576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697027.819099:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697027.819102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145767808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697027.819104:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697027.819107:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697027.819116:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4927 00000100:00080000:3.0:1787697027.819122:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697027.819128:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697027.819129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697027.819133:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697027.819138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145767808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697027.819147:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145768704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697027.819160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145767936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697027.819196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f5187389500 x1874531145767936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145767936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145767936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697027.819245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145768064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697027.819270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f5187389180 x1874531145768064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145768064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145768064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697027.819312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145768192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697027.819328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f518738b800 x1874531145768192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145768192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145768192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697027.819387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697027.819402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f518738bb80 x1874531145768320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145768320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697027.819462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697027.819478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f518738a300 x1874531145768448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145768448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697027.819513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697027.819523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f5187388000 x1874531145768576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145768576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697027.819548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697027.819559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697027.819570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697027/real 1787697027] req@ffff8f5187389f80 x1874531145768704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697027.819582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145768704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697027.819597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697027.819600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697027.819604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145768704:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697029.738595:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697030.762588:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697031.786597:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697032.938467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697032.938478:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697032.938482:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.938493:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938505:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.938513:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697032.938554:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145768832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697032.938585:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697032.938589:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697032.938591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.938596:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938601:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.938606:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.938617:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697032.938625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145768832:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697032.938634:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145768960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697032.938648:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697032.938652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697032.938654:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.938659:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938664:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.938667:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.938673:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697032.938683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697032.938691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145769088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697032.938703:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697032.938707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697032.938709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.938714:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938718:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.938721:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.938728:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697032.938734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f518738ad80 x1874531145768832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697032.938744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145769216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697032.938754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145768832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697032.938758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697032.938764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697032.938766:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697032.938775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.938779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.938783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145768832:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697032.938786:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938793:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.938799:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.938806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697032.938811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145768960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697032.938826:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145769344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697032.938835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697032.938846:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697032.938848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f5187388380 x1874531145768960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697032.938855:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697032.938858:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.938866:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938873:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697032.938875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145768960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697032.938882:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697032.938887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697032.938891:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697032.938895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.938902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145768960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697032.938911:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145769472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697032.938912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145769088:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697032.938930:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697032.938934:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697032.938936:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.938941:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00080000:3.0:1787697032.938946:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697032.938952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697032.938953:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.938962:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697032.938968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f518738b100 x1874531145769088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697032.938979:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145769600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697032.938987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145769088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697032.938996:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697032.938998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697032.939006:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697032.939012:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697032.939013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145769088:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697032.939023:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00100000:1.0:1787697032.939025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145769216:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697032.939030:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.939038:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.939045:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697032.939046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697032.939057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f5187388a80 x1874531145769216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697032.939065:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145769728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697032.939069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145769216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697032.939092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697032.939099:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697032.939101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145769216:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697032.939109:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697032.939112:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697032.939118:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4932 00000100:00100000:1.0:1787697032.939122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145769344:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697032.939123:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697032.939133:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697032.939141:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697032.939144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697032.939155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f518738a680 x1874531145769344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697032.939161:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145769856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697032.939168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145769344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697032.939189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.939195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145769344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697032.939200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697032.939211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697032.939220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f5187388e00 x1874531145769472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697032.939232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145769472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697032.939241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.939247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697032.939254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697032.939270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697032.939281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f518738b480 x1874531145769600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697032.939299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145769600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697032.939307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.939313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697032.939319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697032.939337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697032.939411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f5187389c00 x1874531145769728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697032.939436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145769728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697032.939444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.939452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697032.939468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697032.939484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697032.939492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697032/real 1787697032] req@ffff8f5187388a80 x1874531145769856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697032.939500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145769856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697032.939509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697032.939512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697032.939514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145769856:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697033.834558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697034.858539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697035.882571:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697037.930541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697038.058418:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058427:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697038.058431:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058441:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058453:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058460:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697038.058495:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145769984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697038.058516:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058519:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697038.058520:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058524:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058527:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058530:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058539:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697038.058548:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145770112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697038.058560:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697038.058561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145769984:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697038.058568:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697038.058571:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058577:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058581:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058585:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058592:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697038.058606:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145770240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697038.058616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697038.058620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058623:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058625:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058629:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697038.058634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058636:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697038.058650:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145770368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697038.058660:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058663:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697038.058664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058670:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058674:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058677:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058682:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697038.058700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145770496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697038.058708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f518738b100 x1874531145769984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697038.058713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058722:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697038.058726:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058731:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058736:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697038.058740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145769984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697038.058745:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697038.058759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.058767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697038.058772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145770624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697038.058785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145769984:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697038.058785:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058795:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697038.058797:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058804:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058809:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058814:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697038.058817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770112:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697038.058821:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697038.058835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145770752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697038.058843:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058846:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697038.058847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697038.058849:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058852:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697038.058854:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058857:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697038.058865:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145770880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697038.058868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697038.058880:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697038.058884:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697038.058886:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697038.058888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f5187388380 x1874531145770112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697038.058893:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4937 00000100:00080000:3.0:1787697038.058900:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697038.058912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145770112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697038.058920:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697038.058927:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697038.058940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697038.058941:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145771008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697038.058950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.058960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697038.058971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697038.059004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f518738ad80 x1874531145770240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145770240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697038.059052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697038.059073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f5187389f80 x1874531145770368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145770368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697038.059119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697038.059139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f5187388000 x1874531145770496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145770496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145770496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697038.059195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145770624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697038.059216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f518738a300 x1874531145770624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145770624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145770624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697038.059291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145770752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697038.059313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f518738bb80 x1874531145770752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145770752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145770752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697038.059395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145770880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697038.059417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f518738b800 x1874531145770880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145770880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145770880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697038.059465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145771008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697038.059484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697038.059495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697038/real 1787697038] req@ffff8f5187389180 x1874531145771008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697038.059508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145771008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697038.059517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697038.059520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697038.059524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145771008:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697038.954530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697039.978552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697042.026576:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697043.050531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697043.178670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145771136:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787697043.178754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.178806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace93b80 x1874531145771136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.178834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145771136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.178867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.178876:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.178885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145771136:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697043.178919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697043.178956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.178986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace93100 x1874531145771264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145771264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697043.179050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697043.179080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace90e00 x1874531145771392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145771392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697043.179135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697043.179155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace92680 x1874531145771520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145771520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697043.179220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697043.179239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace90000 x1874531145771648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145771648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145771648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697043.179268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145771776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697043.179288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace93800 x1874531145771776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145771776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145771776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697043.179325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145771904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697043.179335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace90380 x1874531145771904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145771904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145771904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697043.179386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145772032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697043.179398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace90700 x1874531145772032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145772032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145772032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697043.179424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145772160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697043.179433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697043.179439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697043/real 1787697043] req@ffff8f51ace91880 x1874531145772160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697043.179445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145772160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697043.179456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697043.179460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697043.179463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145772160:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697044.074562:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697046.122525:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697047.146525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697048.170503:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697048.298596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145772288:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787697048.298659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.298698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace91180 x1874531145772288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.298718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145772288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.298741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.298747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.298753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145772288:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697048.298777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697048.298801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.298812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace91c00 x1874531145772416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.298840:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145772416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.298856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.298863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.298869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697048.298878:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697048.298899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.298911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace91500 x1874531145772544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.298931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145772544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.298948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.298953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.298959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697048.298965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697048.298982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.298994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace92300 x1874531145772672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.299003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145772672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.299008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.299011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.299013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697048.299017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697048.299026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.299033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace93480 x1874531145772800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.299040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145772800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.299045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.299048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.299050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145772800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697048.299065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145772928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697048.299076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.299082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace92d80 x1874531145772928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.299105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145772928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.299113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.299116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.299119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145772928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697048.299125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145773056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697048.299141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.299151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace91180 x1874531145773056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.299161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145773056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.299179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.299182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.299186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145773056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697048.299191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145773184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697048.299207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.299215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace91880 x1874531145773184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.299221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145773184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.299226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.299228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.299229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145773184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697048.299232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145773312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697048.299240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697048.299245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697048/real 1787697048] req@ffff8f51ace90700 x1874531145773312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697048.299251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145773312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697048.299255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697048.299257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697048.299259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145773312:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697050.218524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697051.242538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697052.266485:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697053.418495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418503:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697053.418506:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418513:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418521:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418526:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697053.418557:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145773440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418576:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418578:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697053.418579:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418583:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418590:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418593:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418600:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697053.418607:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145773568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697053.418617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145773440:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697053.418618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697053.418621:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418627:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418629:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418631:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418634:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697053.418642:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145773696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418649:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697053.418651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418653:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418654:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418656:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418659:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697053.418664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145773824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418669:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697053.418672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418674:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418676:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697053.418679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697053.418681:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418686:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697053.418692:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145773952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697053.418701:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418703:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418707:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418710:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697053.418717:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145774080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697053.418723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f5187389500 x1874531145773440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697053.418724:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697053.418732:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418738:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418742:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418746:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418749:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697053.418758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145773440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697053.418758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145774208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697053.418779:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697053.418781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697053.418783:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418788:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697053.418790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697053.418792:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418796:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697053.418799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145773440:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697053.418803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145774336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697053.418810:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697053.418811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697053.418812:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697053.418815:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4953 00000100:00080000:3.0:1787697053.418816:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697053.418818:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697053.418821:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697053.418826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697053.418828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145774464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697053.418852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.418864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f518738aa00 x1874531145773568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.418876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145773568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.418884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.418887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.418890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697053.418895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697053.418911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.418920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f5187389180 x1874531145773696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.418930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145773696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.418937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.418940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.418943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697053.418948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697053.418962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.418970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f518738b800 x1874531145773824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.418979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145773824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.418986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.418989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.418992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697053.419011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697053.419028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.419037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f518738bb80 x1874531145773952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.419054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145773952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.419061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.419064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.419067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145773952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697053.419072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697053.419086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.419093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f518738a300 x1874531145774080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.419104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145774080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.419112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.419114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.419117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697053.419122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697053.419135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.419143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f5187388000 x1874531145774208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.419152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145774208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.419165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.419168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.419171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697053.419175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697053.419188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.419196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f5187389f80 x1874531145774336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.419206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145774336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.419214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.419216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.419219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697053.419235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697053.419248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697053.419257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697053/real 1787697053] req@ffff8f518738ad80 x1874531145774464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697053.419266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145774464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697053.419274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697053.419276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697053.419280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145774464:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697054.314472:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697055.338514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697056.362611:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697058.410578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697058.538509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.538523:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697058.538529:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.538541:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.538555:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697058.538564:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697058.538615:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145774592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697058.538648:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.538652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697058.538654:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.538659:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.538662:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697058.538667:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.538677:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697058.538692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145774592:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697058.538693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145774720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697058.538709:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.538711:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697058.538713:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.538717:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.538720:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697058.538723:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.538728:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697058.538740:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145774848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697058.538760:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.538766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697058.538767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697058.538771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697058.538773:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.538778:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697058.538782:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.538788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697058.538799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145774976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697058.538808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.538811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697058.538813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.538818:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.538821:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697058.538826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f5187388380 x1874531145774592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697058.538828:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.538835:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697058.538847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145775104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697058.538849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145774592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697058.538870:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697058.538874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697058.538875:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697058.538879:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.538886:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:1.0:1787697058.538890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697058.538891:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697058.538898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145774592:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697058.538899:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.538907:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697058.538932:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145775232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697058.538941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145774720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697058.538944:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.538950:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697058.538952:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.538957:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.538960:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697058.538964:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.538969:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697058.538974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697058.538990:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145775360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697058.538991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f518738b100 x1874531145774720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697058.539014:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697058.539019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145774720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697058.539020:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697058.539024:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.539032:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:1.0:1787697058.539036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697058.539037:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697058.539045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697058.539046:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.539054:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697058.539055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145774720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697058.539066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145774848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697058.539068:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145775488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697058.539079:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697058.539082:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697058.539083:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697058.539087:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4958 00000100:00080000:3.0:1787697058.539090:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697058.539094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697058.539096:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697058.539102:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697058.539107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f5187388a80 x1874531145774848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697058.539114:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145775616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697058.539120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145774848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145774848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697058.539147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145774976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697058.539160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697058.539167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f5187389c00 x1874531145774976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697058.539174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145774976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145774976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697058.539185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145775104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697058.539194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697058.539201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f518738b480 x1874531145775104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697058.539218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145775104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145775104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697058.539250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697058.539264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697058.539271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f5187388e00 x1874531145775232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697058.539279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145775232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697058.539298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697058.539315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697058.539326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f518738a680 x1874531145775360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697058.539343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145775360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697058.539380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697058.539394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697058.539401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f5187388380 x1874531145775488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697058.539408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145775488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697058.539426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697058.539443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697058.539453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697058/real 1787697058] req@ffff8f518738ad80 x1874531145775616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697058.539463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145775616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697058.539468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697058.539470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697058.539472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145775616:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697059.434543:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697060.458560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697062.506544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697063.530511:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697063.658411:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658421:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697063.658425:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658434:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658443:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658451:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697063.658485:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145775744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697063.658509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658512:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697063.658514:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658518:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658522:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658526:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658534:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697063.658542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145775744:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697063.658548:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145775872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697063.658560:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658563:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697063.658565:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658568:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658571:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658575:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658579:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697063.658598:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145776000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697063.658610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697063.658615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697063.658620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697063.658623:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658627:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658631:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658636:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697063.658646:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145776128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697063.658655:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697063.658660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697063.658662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f5187389f80 x1874531145775744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697063.658666:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658670:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658681:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658687:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697063.658689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145775744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.658698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697063.658701:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145776256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697063.658703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.658709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145775744:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697063.658721:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658725:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697063.658726:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658731:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00100000:1.0:1787697063.658733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145775872:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697063.658735:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658750:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697063.658754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658758:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697063.658769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f5187388000 x1874531145775872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697063.658773:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145776384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697063.658796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787697063.658803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145775872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697063.658803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658811:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:1.0:1787697063.658817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697063.658818:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697063.658833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697063.658835:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697063.658839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145775872:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697063.658843:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697063.658846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145776000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697063.658857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145776512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697063.658874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697063.658885:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697063.658889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f518738a300 x1874531145776000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697063.658890:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697063.658893:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658902:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658907:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697063.658910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145776000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697063.658913:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658923:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697063.658935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697063.658935:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145776640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697063.658944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.658951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145776000:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697063.658953:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697063.658959:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697063.658961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145776128:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697063.658962:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697063.658970:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4963 00000100:00080000:3.0:1787697063.658974:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697063.658977:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697063.658981:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697063.658990:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145776768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697063.659001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697063.659015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f518738bb80 x1874531145776128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697063.659026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145776128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.659044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697063.659047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.659050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145776128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697063.659056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145776256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697063.659074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697063.659082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f518738b800 x1874531145776256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697063.659097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145776256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.659104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697063.659107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.659110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145776256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697063.659126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697063.659157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697063.659167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f5187389180 x1874531145776384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697063.659177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145776384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.659184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697063.659187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.659189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697063.659194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697063.659211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697063.659221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f518738aa00 x1874531145776512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697063.659230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145776512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.659238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697063.659240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.659242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697063.659245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697063.659257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697063.659262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f5187389500 x1874531145776640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697063.659268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145776640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.659281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697063.659287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.659290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697063.659295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697063.659313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697063.659323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697063/real 1787697063] req@ffff8f5187389f80 x1874531145776768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697063.659340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145776768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697063.659345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697063.659347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697063.659349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145776768:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697064.554558:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697066.602540:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697067.626536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697068.650522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697068.778493:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778504:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697068.778508:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778518:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778528:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778534:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697068.778567:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145776896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778596:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778599:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697068.778602:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778606:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778610:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778614:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697068.778624:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697068.778638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145777024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697068.778646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145776896:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697068.778652:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697068.778657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778660:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778663:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778667:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697068.778670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697068.778679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145777152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778685:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778687:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697068.778688:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778690:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778692:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778694:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697068.778697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697068.778703:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145777280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778708:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778714:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787697068.778714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697068.778718:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778722:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778726:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778728:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697068.778730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697068.778787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183646a00 x1874531145776896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697068.778795:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145777408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778809:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697068.778813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145776896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697068.778815:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697068.778819:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:2.0:1787697068.778833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697068.778834:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697068.778840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697068.778843:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697068.778848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145776896:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697068.778849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697068.778863:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145777536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778871:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778873:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697068.778876:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697068.778876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697068.778883:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778886:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778888:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697068.778890:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697068.778898:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145777664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778904:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697068.778907:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697068.778910:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787697068.778911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697068.778914:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778918:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697068.778921:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697068.778924:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697068.778932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183645c00 x1874531145777024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697068.778933:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145777792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697068.778965:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697068.778970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145777024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697068.778971:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697068.778976:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697068.778985:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4968 00000100:00080000:1.0:1787697068.778992:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697068.778997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697068.778998:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787697068.779005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697068.779006:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697068.779011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697068.779019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697068.779023:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145777920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697068.779045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183644380 x1874531145777152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145777152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697068.779089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697068.779108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183647b80 x1874531145777280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145777280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697068.779149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697068.779166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183644e00 x1874531145777408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145777408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145777408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697068.779205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697068.779222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183644700 x1874531145777536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145777536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697068.779277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697068.779294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183644000 x1874531145777664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145777664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697068.779335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697068.779351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183647800 x1874531145777792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145777792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697068.779416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697068.779434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697068.779444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697068/real 1787697068] req@ffff8f5183647100 x1874531145777920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697068.779455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145777920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697068.779463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697068.779466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697068.779469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145777920:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697070.698524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697071.722512:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697072.746528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697073.898551:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898560:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697073.898564:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898573:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898584:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898591:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697073.898625:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145778048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697073.898666:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898669:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697073.898672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898676:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898679:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898683:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697073.898700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145778176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697073.898708:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898711:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697073.898712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697073.898716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145778048:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697073.898717:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898722:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697073.898746:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145778304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697073.898755:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898757:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697073.898758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898760:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898763:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898765:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898768:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787697073.898778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697073.898784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145778432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697073.898798:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898801:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697073.898803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898811:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898815:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898819:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697073.898830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f518738ad80 x1874531145778048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697073.898835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145778560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697073.898853:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898856:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697073.898857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697073.898862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145778048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697073.898863:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898869:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898875:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898880:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787697073.898881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.898887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697073.898889:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145778688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697073.898894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145778048:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697073.898900:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898903:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697073.898904:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898907:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898909:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898911:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898913:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697073.898919:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145778816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697073.898922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778176:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697073.898930:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898933:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697073.898935:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898937:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898939:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697073.898941:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898943:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697073.898950:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145778944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697073.898957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697073.898962:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.898965:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697073.898967:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.898969:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4973 00000100:00080000:3.0:1787697073.898972:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697073.898976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f5187388380 x1874531145778176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697073.898977:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697073.898981:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697073.898991:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145779072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697073.898998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145778176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697073.899023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697073.899040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f518738a680 x1874531145778304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145778304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697073.899081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697073.899096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f5187388e00 x1874531145778432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145778432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697073.899134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697073.899148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f518738b480 x1874531145778560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145778560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145778560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697073.899187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145778688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697073.899201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f5187389c00 x1874531145778688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145778688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145778688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697073.899264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145778816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697073.899281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f5187388a80 x1874531145778816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145778816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145778816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697073.899327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145778944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697073.899343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f518738b100 x1874531145778944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145778944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145778944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697073.899423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145779072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697073.899441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697073.899451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697073/real 1787697073] req@ffff8f518738ad80 x1874531145779072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697073.899463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145779072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697073.899470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697073.899473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697073.899477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145779072:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697074.794544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697075.818544:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697076.842519:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697078.890538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697079.018494:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018506:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697079.018510:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018522:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.018534:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018542:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697079.018611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145779200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697079.018638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018642:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697079.018644:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018649:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.018652:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018656:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697079.018666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697079.018679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145779328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697079.018690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018694:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697079.018695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018699:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.018702:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018711:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697079.018713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145779200:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697079.018716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697079.018733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145779456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697079.018744:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018748:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697079.018750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018754:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.018757:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018761:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697079.018767:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697079.018777:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145779584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697079.018787:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018790:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697079.018792:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018798:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4978 00000800:00000100:2.0:1787697079.018800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697079.018803:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018808:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697079.018813:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697079.018827:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145779712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697079.018836:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018840:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697079.018842:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018846:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.018849:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018853:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697079.018858:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697079.018864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183646680 x1874531145779200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697079.018874:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145779840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697079.018899:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.018903:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787697079.018907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145779200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697079.018909:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.018923:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.018933:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.018940:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787697079.018944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697079.018945:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787697079.018954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697079.018963:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145779968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697079.018964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145779200:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697079.018984:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697079.018986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697079.018991:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697079.018994:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.019009:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.019015:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697079.019019:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697079.019024:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787697079.019029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697079.019043:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145780096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697079.019053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183647480 x1874531145779328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697079.019059:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697079.019069:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697079.019072:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697079.019080:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4978 00000100:00080000:1.0:1787697079.019084:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697079.019086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145779328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697079.019091:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697079.019099:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697079.019110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697079.019117:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145780224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697079.019134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697079.019170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697079.019192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183646d80 x1874531145779456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145779456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697079.019232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697079.019248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183645180 x1874531145779584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145779584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697079.019283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697079.019298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183647100 x1874531145779712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145779712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145779712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697079.019335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145779840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697079.019391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183647800 x1874531145779840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145779840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145779840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697079.019436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145779968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697079.019454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183644000 x1874531145779968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145779968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145779968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697079.019495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145780096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697079.019515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183644700 x1874531145780096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145780096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145780096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697079.019572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145780224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697079.019594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697079.019605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697079/real 1787697079] req@ffff8f5183644e00 x1874531145780224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697079.019617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145780224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697079.019624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697079.019628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697079.019631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145780224:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697079.914509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697080.938599:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697082.986535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697084.010561:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697084.138508:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138518:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697084.138522:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138531:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138550:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138558:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697084.138599:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145780352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697084.138626:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138630:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697084.138633:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138638:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138650:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138656:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697084.138679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145780352:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697084.138685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145780480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697084.138699:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138703:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697084.138705:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138709:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138714:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138718:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697084.138738:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145780608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697084.138746:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138749:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697084.138750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138754:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138757:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138759:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138763:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697084.138772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145780736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697084.138778:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138780:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697084.138782:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138784:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138786:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138788:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138792:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697084.138800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145780864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697084.138806:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697084.138808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697084.138810:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697084.138813:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138821:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138824:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138827:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138830:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697084.138841:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145780992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697084.138847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697084.138852:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138858:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00100000:2.0:1787697084.138859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697084.138860:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138865:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697084.138902:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145781120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697084.138909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780736:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697084.138911:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138918:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697084.138919:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138923:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138925:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697084.138927:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.138931:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697084.138958:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145781248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697084.138963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780864:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697084.138978:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697084.138982:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697084.138985:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697084.138990:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4983 00000100:00080000:3.0:1787697084.138995:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697084.138998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145780992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697084.139003:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697084.139010:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697084.139022:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145781376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697084.139030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145781120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697084.139060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145781248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697084.139080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145781376:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697085.034553:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697087.082740:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787697087.210730:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787697087.210775:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787697087.210792:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210839:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210857:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210869:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210879:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210889:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787697087.210894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f518738ad80 x1874531145781376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697087.210904:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210918:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697087.210931:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787697087.210936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145781376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.210950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.210956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.210961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145781376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697087.210969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f518738b800 x1874531145780864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.210981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145780864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.210994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.210999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697087.211010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f518738bb80 x1874531145780992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145780992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145780992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697087.211045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f518738a300 x1874531145781120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145781120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145781120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697087.211077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f5187389500 x1874531145780480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145780480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697087.211115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f518738aa00 x1874531145780608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145780608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697087.211146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f5187389180 x1874531145780736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145780736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145780736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697087.211176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f5187389f80 x1874531145780352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145780352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145780352:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697087.211234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697084/real 1787697087] req@ffff8f5187388000 x1874531145781248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697087.211245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145781248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697087.211255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697087.211259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697087.211262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145781248:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697088.106537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697089.130552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697089.258471:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697089.258490:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697089.258493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.258502:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258511:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697089.258516:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697089.258553:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145781504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697089.258584:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697089.258588:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697089.258590:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.258595:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697089.258604:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697089.258614:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697089.258629:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145781632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697089.258642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697089.258645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145781504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697089.258647:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697089.258651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.258658:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258662:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697089.258666:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697089.258671:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697089.258685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145781760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697089.258695:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697089.258699:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697089.258701:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.258706:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258710:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697089.258715:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697089.258720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787697089.258723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697089.258736:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145781888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697089.258747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697089.258751:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697089.258753:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.258758:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258762:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697089.258767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704f800 x1874531145781504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697089.258771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697089.258779:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697089.258801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145781504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697089.258805:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145782016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697089.258820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697089.258825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697089.258828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697089.258831:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787697089.258837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145781504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697089.258837:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.258849:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258854:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697089.258858:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697089.258863:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697089.258870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145781632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697089.258879:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145782144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697089.258891:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697089.258895:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697089.258898:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697089.258903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697089.258906:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.258913:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697089.258931:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697089.258935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704e300 x1874531145781632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697089.258939:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697089.258959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145781632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697089.258961:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145782272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697089.258988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697089.258990:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697089.258995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697089.258997:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697089.259002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145781632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697089.259002:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697089.259012:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.259018:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697089.259019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145781760:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697089.259026:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697089.259033:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697089.259047:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145782400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697089.259051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697089.259063:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697089.259068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704f480 x1874531145781760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697089.259069:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697089.259073:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697089.259088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145781760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697089.259090:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4988 00000100:00080000:2.0:1787697089.259098:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697089.259106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697089.259106:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697089.259115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787697089.259116:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697089.259122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145781760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697089.259135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145781888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697089.259136:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145782528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697089.259163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697089.259179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704d500 x1874531145781888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697089.259193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145781888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697089.259201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697089.259205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697089.259209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145781888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697089.259215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145782016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697089.259236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697089.259247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704fb80 x1874531145782016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697089.259261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145782016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697089.259269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697089.259273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697089.259276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145782016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697089.259296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697089.259319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697089.259331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704dc00 x1874531145782144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697089.259345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145782144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697089.259394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697089.259399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697089.259403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697089.259410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697089.259432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697089.259454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704ed80 x1874531145782272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697089.259468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145782272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697089.259484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697089.259487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697089.259491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697089.259498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697089.259521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697089.259532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704d180 x1874531145782400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697089.259544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145782400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697089.259552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697089.259555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697089.259559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697089.259565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697089.259585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697089.259596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697089/real 1787697089] req@ffff8f518704f800 x1874531145782528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697089.259607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145782528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697089.259614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697089.259617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697089.259620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145782528:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697091.178496:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697092.202534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697093.226489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697094.378447:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697094.378455:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697094.378459:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378467:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00080000:1.0:1787697094.378476:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697094.378482:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697094.378516:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145782656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697094.378546:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697094.378551:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697094.378553:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378558:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00080000:1.0:1787697094.378562:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697094.378567:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.378578:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697094.378586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145782656:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697094.378595:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145782784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697094.378608:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697094.378611:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697094.378613:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378616:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00080000:1.0:1787697094.378620:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697094.378624:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.378628:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697094.378637:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145782912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697094.378645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697094.378647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697094.378651:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697094.378652:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378657:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00080000:1.0:1787697094.378670:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697094.378674:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.378680:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697094.378694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183647b80 x1874531145782656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697094.378696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145783040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697094.378729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697094.378733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145782656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697094.378735:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697094.378739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378750:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00080000:2.0:1787697094.378751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697094.378759:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697094.378759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.378766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145782656:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697094.378769:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.378777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697094.378789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145782784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697094.378795:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145783168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697094.378807:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787697094.378812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697094.378816:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697094.378820:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378825:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00100000:2.0:1787697094.378828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183644380 x1874531145782784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697094.378831:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697094.378841:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697094.378846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145782784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697094.378847:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787697094.378858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.378862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.378867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145782784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697094.378869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145783296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697094.378873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145782912:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697094.378888:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697094.378893:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787697094.378894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697094.378898:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378907:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00100000:2.0:1787697094.378907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183645c00 x1874531145782912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697094.378915:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697094.378921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145782912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697094.378923:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.378933:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787697094.378943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.378947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.378952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145782912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697094.378954:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145783424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697094.378960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145783040:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697094.378972:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697094.378977:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697094.378980:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.378986:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4994 00000800:00000100:2.0:1787697094.378991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697094.378992:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697094.379000:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.379006:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697094.379009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183646a00 x1874531145783040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697094.379022:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145783552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697094.379027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145783040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697094.379047:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697094.379048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.379055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697094.379057:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787697094.379060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145783040:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697094.379062:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697094.379070:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4994 00000100:00080000:1.0:1787697094.379074:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697094.379079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145783168:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697094.379080:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697094.379090:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787697094.379103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697094.379107:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145783680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697094.379115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183647b80 x1874531145783168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697094.379130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145783168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697094.379136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.379137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.379140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145783168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697094.379145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697094.379157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697094.379166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183644e00 x1874531145783296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697094.379174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145783296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697094.379180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.379181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.379183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697094.379187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697094.379196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697094.379202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183644700 x1874531145783424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697094.379209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145783424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697094.379214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.379215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.379217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697094.379221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697094.379230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697094.379236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183644000 x1874531145783552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697094.379244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145783552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697094.379248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.379250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.379252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697094.379261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697094.379270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697094.379276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697094/real 1787697094] req@ffff8f5183647800 x1874531145783680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697094.379283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145783680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697094.379288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697094.379304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697094.379306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145783680:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697095.274562:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697096.298571:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697097.322530:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697099.370521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697099.498487:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498497:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697099.498501:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498511:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498522:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498528:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697099.498563:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145783808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697099.498594:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498598:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697099.498600:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498604:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498607:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697099.498623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697099.498636:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145783936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697099.498647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145783808:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697099.498650:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498659:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697099.498661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498665:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498668:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498673:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697099.498678:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697099.498690:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145784064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697099.498699:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498702:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697099.498704:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498708:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498712:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498715:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697099.498720:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787697099.498726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697099.498735:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145784192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697099.498746:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498750:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697099.498752:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498757:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498760:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498764:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697099.498770:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697099.498778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183647100 x1874531145783808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697099.498783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145784320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697099.498809:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498813:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697099.498818:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697099.498819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145783808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697099.498824:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498831:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498838:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787697099.498839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697099.498845:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787697099.498846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.498852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145783808:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697099.498861:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145784448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697099.498872:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498876:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787697099.498878:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145783936:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697099.498880:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498889:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:1.0:1787697099.498893:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.498897:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697099.498902:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787697099.498909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697099.498916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145784576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697099.498923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183645180 x1874531145783936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697099.498934:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697099.498939:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787697099.498941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145783936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697099.498942:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.498953:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00080000:2.0:1787697099.498954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697099.498957:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697099.498960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697099.498966:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697099.498967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145783936:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697099.498976:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697099.498978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145784064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697099.498993:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145784704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697099.499003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697099.499012:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697099.499016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183646d80 x1874531145784064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697099.499018:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697099.499022:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697099.499032:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 4999 00000100:00100000:2.0:1787697099.499036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145784064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697099.499038:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697099.499047:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787697099.499048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697099.499054:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697099.499054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145784064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697099.499067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145784192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697099.499072:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145784832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697099.499085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697099.499095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183647480 x1874531145784192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697099.499105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145784192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697099.499110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697099.499114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145784192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697099.499130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145784320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697099.499141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697099.499148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183646680 x1874531145784320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697099.499155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145784320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697099.499165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697099.499167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145784320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697099.499173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697099.499184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697099.499190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183647800 x1874531145784448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697099.499198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145784448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697099.499233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697099.499236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697099.499253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697099.499269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697099.499277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183644000 x1874531145784576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697099.499285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145784576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697099.499290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697099.499293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697099.499298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697099.499308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697099.499313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183644700 x1874531145784704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697099.499324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145784704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697099.499338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697099.499342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697099.499350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697099.499399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697099.499412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697099/real 1787697099] req@ffff8f5183644e00 x1874531145784832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697099.499422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145784832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697099.499430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697099.499433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697099.499436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145784832:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697100.394540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697101.418493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697103.466539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697104.490602:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697104.618480:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618492:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697104.618496:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697104.618507:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00080000:2.0:1787697104.618518:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.618525:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697104.618582:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145784960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697104.618613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697104.618620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697104.618625:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00080000:2.0:1787697104.618630:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.618635:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.618656:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697104.618657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145784960:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697104.618689:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145785088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697104.618703:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697104.618709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697104.618713:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5004 00000800:00000100:1.0:1787697104.618718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697104.618719:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.618726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.618732:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697104.618744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145785216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697104.618754:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697104.618760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697104.618763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00080000:2.0:1787697104.618768:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.618772:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.618789:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697104.618793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704ea00 x1874531145784960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697104.618812:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145785344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697104.618823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145784960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697104.618826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618836:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697104.618839:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697104.618841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.618847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697104.618847:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00100000:1.0:1787697104.618855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145784960:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697104.618856:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.618865:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.618871:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697104.618883:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697104.618886:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145785472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697104.618900:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618904:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697104.618906:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697104.618911:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00080000:2.0:1787697104.618915:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697104.618916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697104.618923:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.618930:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697104.618939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704d880 x1874531145785088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697104.618946:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145785600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697104.618956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145785088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.618965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.618967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.618971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785088:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697104.618971:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.618981:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697104.618985:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697104.618990:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00100000:1.0:1787697104.618992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697104.618995:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.619003:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.619015:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697104.619031:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145785728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697104.619040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697104.619050:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.619054:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697104.619056:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697104.619060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704c700 x1874531145785216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697104.619065:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00080000:2.0:1787697104.619071:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.619079:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697104.619085:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697104.619091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145785216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697104.619100:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145785856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697104.619106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697104.619139:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697104.619147:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697104.619149:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697104.619151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785344:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697104.619159:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5004 00000100:00080000:2.0:1787697104.619166:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697104.619170:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697104.619184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787697104.619185:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697104.619199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704c380 x1874531145785344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697104.619204:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145785984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697104.619227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145785344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.619238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697104.619273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697104.619290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697104.619298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704c000 x1874531145785472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697104.619311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145785472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.619317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145785472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697104.619325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697104.619335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697104.619341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704e680 x1874531145785600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697104.619392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145785600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.619408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697104.619423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697104.619461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697104.619471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704ce00 x1874531145785728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697104.619481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145785728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.619496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697104.619509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697104.619529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697104.619548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704df80 x1874531145785856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697104.619564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145785856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.619571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697104.619582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697104.619601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697104.619612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697104/real 1787697104] req@ffff8f518704ea00 x1874531145785984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697104.619624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145785984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697104.619633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697104.619636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697104.619640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145785984:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697105.514679:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697107.562548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697108.586565:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697109.610572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697109.738440:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738448:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697109.738451:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738459:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738469:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738475:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697109.738512:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145786112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697109.738550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697109.738554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697109.738588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145786112:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697109.738598:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738604:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738609:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738631:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697109.738644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145786240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697109.738649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697109.738659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738662:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697109.738664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738667:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738671:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738675:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697109.738687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145786368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697109.738694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697109.738698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738703:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00100000:1.0:1787697109.738704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704f800 x1874531145786112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697109.738705:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738710:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697109.738728:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145786496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697109.738735:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697109.738738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145786112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697109.738738:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697109.738741:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738747:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738751:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738756:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697109.738763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697109.738769:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145786624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697109.738772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.738782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145786112:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697109.738784:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738792:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697109.738794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738798:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738800:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738803:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738805:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697109.738812:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145786752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697109.738817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786240:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697109.738821:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738824:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697109.738825:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738831:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738833:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738837:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697109.738845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145786880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697109.738852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697109.738855:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738858:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697109.738860:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738864:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738866:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738869:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697109.738874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704d180 x1874531145786240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697109.738874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697109.738883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145787008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697109.738889:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697109.738894:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697109.738896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697109.738904:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5009 00000100:00080000:2.0:1787697109.738908:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697109.738912:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697109.738915:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697109.738921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145786240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697109.738923:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145787136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697109.738956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.738961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.738966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697109.738974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697109.739000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704ed80 x1874531145786368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145786368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697109.739095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697109.739113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704dc00 x1874531145786496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145786496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697109.739144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697109.739155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704fb80 x1874531145786624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145786624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145786624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697109.739194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145786752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697109.739205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704d500 x1874531145786752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145786752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145786752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697109.739240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145786880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697109.739258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704f480 x1874531145786880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145786880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145786880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697109.739298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145787008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697109.739311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704e300 x1874531145787008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145787008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145787008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697109.739338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145787136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697109.739347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697109.739376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697109/real 1787697109] req@ffff8f518704f800 x1874531145787136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697109.739385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145787136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697109.739390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697109.739392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697109.739395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145787136:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697111.658579:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697112.682529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697113.706554:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697114.858496:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858503:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697114.858506:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858513:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858521:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858525:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697114.858554:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145787264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697114.858574:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858576:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697114.858577:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858580:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858583:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858587:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858597:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697114.858610:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145787392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697114.858613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145787264:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697114.858629:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858632:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697114.858634:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858637:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858640:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858643:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858646:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697114.858655:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145787520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697114.858660:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858662:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697114.858664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858666:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858671:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858673:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858676:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787697114.858683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697114.858684:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145787648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697114.858691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858694:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697114.858695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858698:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858702:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858705:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697114.858711:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145787776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697114.858716:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858718:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697114.858719:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858722:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858724:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697114.858736:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145787904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697114.858742:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858744:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787697114.858744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183647b80 x1874531145787264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697114.858747:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858755:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858761:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858764:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858767:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697114.858776:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145788032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697114.858776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145787264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697114.858799:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858804:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697114.858805:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697114.858807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697114.858808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858813:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697114.858816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697114.858817:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858823:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697114.858826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145787264:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697114.858831:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145788160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697114.858837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697114.858840:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697114.858841:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697114.858843:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5014 00000100:00080000:1.0:1787697114.858845:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697114.858847:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697114.858849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697114.858856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697114.858857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145788288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697114.858897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.858911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183646a00 x1874531145787392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.858924:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145787392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.858932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.858935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.858954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697114.858960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697114.858980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.858992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183645c00 x1874531145787520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145787520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697114.859023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697114.859041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.859051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183644380 x1874531145787648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145787648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697114.859087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697114.859104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.859114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183644e00 x1874531145787776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145787776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145787776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697114.859142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145787904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697114.859157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.859168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183644700 x1874531145787904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145787904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145787904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697114.859203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145788032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697114.859219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.859228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183644000 x1874531145788032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145788032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145788032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697114.859270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145788160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697114.859289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.859299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183647800 x1874531145788160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145788160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145788160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697114.859342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145788288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697114.859378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697114.859389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697114/real 1787697114] req@ffff8f5183646680 x1874531145788288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697114.859400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145788288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697114.859407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697114.859409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697114.859412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145788288:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697115.754524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697116.778482:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697117.802606:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697119.850556:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697119.978466:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978474:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697119.978478:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978485:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978499:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697119.978528:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145788416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697119.978552:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978555:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978557:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978560:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978568:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697119.978575:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145788544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978582:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978585:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697119.978587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697119.978589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145788416:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697119.978591:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978594:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978598:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978602:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697119.978607:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145788672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978615:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697119.978616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978618:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978620:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978622:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697119.978631:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145788800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978635:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978638:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697119.978639:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978642:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978646:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978649:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978653:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697119.978658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697119.978667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145788928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978677:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697119.978683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978686:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978689:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978693:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697119.978704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145789056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978712:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697119.978714:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697119.978718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704ea00 x1874531145788416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697119.978718:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978727:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978734:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697119.978745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145789184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697119.978750:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697119.978754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145788416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697119.978754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697119.978757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978766:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978771:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978774:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697119.978775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697119.978777:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697119.978784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697119.978787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145789312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697119.978793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145788416:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697119.978796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697119.978802:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697119.978803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697119.978807:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5019 00000100:00080000:2.0:1787697119.978808:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697119.978810:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697119.978813:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697119.978819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145789440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697119.978825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697119.978866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.978878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704df80 x1874531145788544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.978891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145788544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.978899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.978902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.978905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697119.978911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697119.978929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.978939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704ce00 x1874531145788672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.978949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145788672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.978957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.978960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.978963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697119.978969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697119.978989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.979000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704e680 x1874531145788800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.979011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145788800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.979018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.979021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.979025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697119.979031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697119.979048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.979058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704c000 x1874531145788928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.979068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145788928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.979076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.979079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.979082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145788928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697119.979087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697119.979126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.979137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704c380 x1874531145789056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.979149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145789056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.979163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.979166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.979168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697119.979174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697119.979191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.979201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704c700 x1874531145789184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.979212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145789184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.979220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.979223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.979226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697119.979231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697119.979248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.979258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704d880 x1874531145789312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.979269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145789312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.979276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.979279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.979283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697119.979303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697119.979321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697119.979332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697119/real 1787697119] req@ffff8f518704ea00 x1874531145789440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697119.979344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145789440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697119.979373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697119.979377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697119.979380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145789440:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697120.874492:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697121.898544:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697123.946505:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697124.970498:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697125.098470:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697125.098480:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098488:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:1.0:1787697125.098497:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098502:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697125.098531:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145789568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697125.098550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697125.098554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098557:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:1.0:1787697125.098560:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098563:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098570:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697125.098578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145789696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697125.098582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145789568:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697125.098586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098591:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697125.098592:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098595:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:1.0:1787697125.098597:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098601:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697125.098608:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145789824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697125.098613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098622:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697125.098624:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098631:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:1.0:1787697125.098637:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098640:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098645:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787697125.098646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697125.098659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145789952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697125.098672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098674:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697125.098675:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098678:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:1.0:1787697125.098681:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098683:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098687:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697125.098690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183647480 x1874531145789568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697125.098698:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145790080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697125.098705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145789568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697125.098706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697125.098716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098723:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:2.0:1787697125.098724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697125.098725:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697125.098730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697125.098731:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697125.098737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145789568:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697125.098744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145790208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697125.098750:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098753:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697125.098754:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098758:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00100000:2.0:1787697125.098758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145789696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697125.098773:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098777:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697125.098790:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145790336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697125.098797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697125.098803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697125.098806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697125.098807:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098811:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00100000:2.0:1787697125.098812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183646d80 x1874531145789696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697125.098814:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697125.098818:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697125.098829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145789696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697125.098833:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145790464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697125.098842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697125.098845:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697125.098848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697125.098851:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787697125.098854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145789696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697125.098855:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697125.098861:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5024 00000100:00080000:1.0:1787697125.098863:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697125.098864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145789824:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697125.098867:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697125.098871:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697125.098880:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145790592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697125.098885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.098895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183645180 x1874531145789824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.098903:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145789824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.098912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.098914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.098916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145789824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697125.098919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145789952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697125.098929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.098934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183647100 x1874531145789952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.098940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145789952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.098945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.098946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.098948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145789952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697125.098951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145790080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697125.098969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.098975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183646680 x1874531145790080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.098981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145790080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.098986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.098987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.098990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145790080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697125.098993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697125.099002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.099010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183647800 x1874531145790208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.099019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145790208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.099027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.099030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.099032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697125.099050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697125.099069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.099080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183644000 x1874531145790336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.099091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145790336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.099097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.099101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.099103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697125.099109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697125.099125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.099136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183644700 x1874531145790464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.099146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145790464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.099153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.099156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.099159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697125.099164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697125.099179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697125.099189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697125/real 1787697125] req@ffff8f5183644e00 x1874531145790592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697125.099199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145790592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697125.099206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697125.099209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697125.099212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145790592:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697125.994482:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697128.042498:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697129.066543:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697130.090495:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697130.218415:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218425:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697130.218429:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218439:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218449:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697130.218457:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697130.218491:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145790720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697130.218516:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218520:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697130.218522:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218526:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218530:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697130.218535:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697130.218544:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697130.218550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145790720:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697130.218559:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145790848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697130.218571:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218574:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697130.218576:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218579:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218583:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697130.218587:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697130.218591:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697130.218603:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145790976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697130.218604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697130.218615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697130.218620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218625:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218628:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697130.218632:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697130.218636:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697130.218645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183644380 x1874531145790720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697130.218647:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145791104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697130.218683:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218688:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787697130.218690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145790720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697130.218691:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218700:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:2.0:1787697130.218704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697130.218705:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697130.218709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697130.218713:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697130.218716:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145790720:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697130.218719:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697130.218735:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145791232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697130.218737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145790848:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697130.218753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218757:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697130.218759:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787697130.218763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697130.218767:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218772:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697130.218778:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697130.218779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183645c00 x1874531145790848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697130.218783:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697130.218797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145791360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697130.218802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145790848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697130.218811:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697130.218821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697130.218821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.218831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697130.218832:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218838:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697130.218839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145790848:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697130.218844:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697130.218849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697130.218855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145790976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697130.218859:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145791488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697130.218869:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697130.218872:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697130.218874:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218878:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5029 00000800:00000100:2.0:1787697130.218883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697130.218884:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697130.218891:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697130.218896:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697130.218901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183646a00 x1874531145790976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697130.218908:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145791616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697130.218920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697130.218920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145790976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697130.218926:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697130.218928:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697130.218936:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5029 00000100:00080000:1.0:1787697130.218939:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697130.218939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697130.218946:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787697130.218948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697130.218952:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697130.218954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145790976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697130.218965:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145791744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697130.218966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145791104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697130.218988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697130.218998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183647b80 x1874531145791104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697130.219005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145791104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697130.219010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.219012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697130.219014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145791104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697130.219024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145791232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697130.219033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697130.219038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183644e00 x1874531145791232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697130.219044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145791232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697130.219049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.219050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697130.219052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145791232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697130.219055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697130.219063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697130.219069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183644700 x1874531145791360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697130.219076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145791360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697130.219081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.219082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697130.219084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697130.219086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697130.219095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697130.219100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183644000 x1874531145791488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697130.219107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145791488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697130.219111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.219112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697130.219114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697130.219117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697130.219129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697130.219138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183647800 x1874531145791616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697130.219149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145791616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697130.219163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.219166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697130.219168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697130.219174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697130.219189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697130.219198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697130/real 1787697130] req@ffff8f5183646680 x1874531145791744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697130.219208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145791744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697130.219214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697130.219217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697130.219220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145791744:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697132.138510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697133.162535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697134.186547:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697135.338471:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338478:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697135.338481:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338489:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338499:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338504:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697135.338535:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145791872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697135.338555:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697135.338559:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338562:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338564:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338566:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338574:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697135.338581:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145792000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697135.338599:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697135.338601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145791872:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697135.338602:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697135.338605:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338610:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338613:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338615:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338618:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697135.338628:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145792128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697135.338634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338636:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697135.338637:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338639:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338651:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338655:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787697135.338659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338662:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697135.338676:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145792256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697135.338685:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338688:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697135.338691:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00100000:2.0:1787697135.338697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f518738b100 x1874531145791872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697135.338700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338712:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697135.338717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145791872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697135.338718:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697135.338733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697135.338733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145792384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697135.338743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697135.338745:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787697135.338750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145791872:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697135.338751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338758:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338760:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338763:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338765:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697135.338772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145792512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697135.338778:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697135.338779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792000:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697135.338784:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697135.338787:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338790:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338792:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338794:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697135.338806:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145792640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697135.338809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697135.338817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697135.338822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697135.338824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f5187388a80 x1874531145792000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697135.338827:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00080000:3.0:1787697135.338831:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338836:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338840:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697135.338842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145792000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697135.338849:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145792768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697135.338852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.338860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697135.338860:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697135.338866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787697135.338867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792000:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697135.338869:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697135.338876:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5034 00000100:00100000:2.0:1787697135.338880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792128:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697135.338881:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697135.338886:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697135.338889:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697135.338898:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145792896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697135.338903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.338915:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f5187389c00 x1874531145792128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.338923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145792128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.338928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.338930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.338932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697135.338935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697135.338945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.338950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f518738b480 x1874531145792256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.338957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145792256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.338961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.338963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.338964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697135.338967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697135.338976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.338981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f5187388e00 x1874531145792384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.338987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145792384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.338991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.338993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.338994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145792384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697135.339004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697135.339013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.339019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f518738a680 x1874531145792512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.339033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145792512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.339042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.339044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.339046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697135.339049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787697135.339059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.339064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f5187388380 x1874531145792640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.339071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145792640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.339075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.339077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.339079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697135.339082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697135.339090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.339095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f518738b100 x1874531145792768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.339101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145792768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.339105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.339106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.339109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787697135.339112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787697135.339120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697135.339125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697135/real 1787697135] req@ffff8f5187388000 x1874531145792896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697135.339130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145792896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697135.339135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697135.339136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697135.339138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145792896:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697136.234509:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697137.258593:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697138.282557:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697140.330472:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697140.458452:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458462:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697140.458466:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697140.458474:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458484:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697140.458490:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697140.458530:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145793024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697140.458551:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458554:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697140.458556:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697140.458560:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458563:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697140.458568:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697140.458579:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697140.458594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145793024:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697140.458595:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145793152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697140.458608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697140.458611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697140.458614:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458617:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697140.458620:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697140.458625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697140.458637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145793280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697140.458647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697140.458650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697140.458653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697140.458655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697140.458659:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697140.458668:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697140.458672:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697140.458681:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145793408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697140.458687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697140.458690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697140.458693:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458695:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697140.458699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704f800 x1874531145793024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697140.458699:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697140.458706:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697140.458718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145793536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697140.458724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145793024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697140.458728:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458735:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697140.458738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697140.458742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697140.458743:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458747:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697140.458748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697140.458751:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697140.458754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145793024:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787697140.458756:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697140.458765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145793664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697140.458773:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697140.458775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793152:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697140.458777:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697140.458780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697140.458783:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458785:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697140.458788:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697140.458792:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697140.458796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697140.458804:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145793792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697140.458808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704e300 x1874531145793152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697140.458817:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458823:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697140.458824:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697140.458827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145793152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697140.458830:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458835:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697140.458839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697140.458839:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697140.458844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787697140.458845:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697140.458849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697140.458854:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145793920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697140.458856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697140.458867:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697140.458870:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697140.458872:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697140.458875:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697140.458879:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5040 00000100:00080000:2.0:1787697140.458884:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697140.458889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704f480 x1874531145793280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697140.458891:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697140.458899:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697140.458907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145793280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697140.458914:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145794048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697140.458922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.458927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.458931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697140.458936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697140.458958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697140.458966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704d500 x1874531145793408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697140.458973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145793408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697140.458978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.458980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.458982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697140.458985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697140.458994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697140.458999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704fb80 x1874531145793536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697140.459005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145793536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697140.459009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.459011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.459012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145793536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697140.459022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145793664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697140.459031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697140.459037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704dc00 x1874531145793664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697140.459043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145793664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697140.459048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.459050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.459052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145793664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697140.459055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145793792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697140.459063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697140.459067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704ed80 x1874531145793792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697140.459073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145793792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697140.459078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.459079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.459081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145793792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697140.459084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145793920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697140.459098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697140.459107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704d180 x1874531145793920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697140.459118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145793920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697140.459125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.459128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.459132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145793920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697140.459135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145794048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697140.459177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697140.459188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697140/real 1787697140] req@ffff8f518704f800 x1874531145794048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697140.459199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145794048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697140.459214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697140.459218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697140.459221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145794048:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697141.354517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697142.378492:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697144.426578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697145.450518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697145.578471:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578478:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697145.578482:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697145.578490:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578499:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578504:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697145.578552:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145794176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697145.578575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578577:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697145.578579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697145.578582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578586:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578593:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697145.578601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145794304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697145.578609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697145.578613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697145.578617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145794176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697145.578618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578627:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697145.578638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145794432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697145.578648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697145.578652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697145.578655:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578663:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578665:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578668:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697145.578671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697145.578680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145794560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697145.578686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697145.578690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697145.578692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578694:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578696:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697145.578705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704ea00 x1874531145794176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697145.578711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145794688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697145.578740:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697145.578743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145794176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697145.578746:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697145.578750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697145.578756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697145.578758:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:3.0:1787697145.578761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.578767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145794176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697145.578768:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578775:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578780:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697145.578785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697145.578794:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145794816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697145.578803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697145.578806:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578810:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697145.578813:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697145.578813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704d880 x1874531145794304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697145.578822:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00100000:3.0:1787697145.578828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145794304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697145.578829:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578838:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787697145.578843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697145.578846:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697145.578849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.578855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697145.578863:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145794944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697145.578864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794432:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697145.578875:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578877:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697145.578878:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697145.578884:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578887:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578890:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578895:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697145.578900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697145.578904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145795072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697145.578912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704c700 x1874531145794432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697145.578918:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697145.578924:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697145.578927:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697145.578933:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5045 00000100:00080000:2.0:1787697145.578936:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697145.578942:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697145.578945:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697145.578949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145794432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.578963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.578967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697145.578969:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145795200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697145.578969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697145.578980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697145.579003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697145.579011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704c380 x1874531145794560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697145.579018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145794560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.579022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.579024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.579026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697145.579028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697145.579037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697145.579044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704c000 x1874531145794688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697145.579054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145794688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.579061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.579064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.579076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145794688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697145.579093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145794816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697145.579108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697145.579118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704e680 x1874531145794816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697145.579129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145794816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.579137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.579140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.579143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145794816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697145.579149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145794944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697145.579165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697145.579171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704ce00 x1874531145794944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697145.579178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145794944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.579182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.579184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.579186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145794944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697145.579189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145795072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697145.579197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697145.579203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704df80 x1874531145795072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697145.579209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145795072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.579214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.579215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.579217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145795072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697145.579220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145795200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697145.579228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697145.579233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697145/real 1787697145] req@ffff8f518704ea00 x1874531145795200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697145.579240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145795200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697145.579244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697145.579246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697145.579247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145795200:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697146.474517:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697148.522541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697149.546503:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697150.570507:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697150.698424:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698431:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697150.698434:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698441:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698449:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698455:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697150.698484:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145795328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697150.698502:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698505:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697150.698506:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698509:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698511:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698513:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698521:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697150.698529:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145795456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697150.698536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145795328:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697150.698539:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698544:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697150.698545:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698548:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698549:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698551:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698554:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697150.698562:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145795584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697150.698567:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698569:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697150.698571:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698574:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698583:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698585:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698588:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697150.698594:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145795712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697150.698600:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698602:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697150.698604:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698606:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698608:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698610:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698613:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697150.698621:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145795840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697150.698629:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787697150.698634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795456:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697150.698636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698644:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698647:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698652:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697150.698660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697150.698673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145795968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697150.698678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795712:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697150.698692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698695:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697150.698697:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697150.698698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795840:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697150.698702:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698707:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698710:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697150.698719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145795968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697150.698727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145796096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697150.698735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698738:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697150.698738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698740:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698742:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698744:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697150.698746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145796096:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787697150.698749:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697150.698759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145796224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697150.698767:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697150.698771:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787697150.698771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145796224:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697150.698774:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697150.698779:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5050 00000100:00080000:2.0:1787697150.698782:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697150.698784:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697150.698788:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697150.698796:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145796352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697150.698823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145796352:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697152.618571:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697153.642528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697153.770812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704ea00 x1874531145796352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.770840:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145796352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.770852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.770857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.770860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145796352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.770867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704e300 x1874531145796224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.770879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145796224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.770888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.770892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.770894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145796224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.770899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704f480 x1874531145796096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.770911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145796096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.770918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.770921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.770923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145796096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697153.770942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704d500 x1874531145795968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.770955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145795968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.770964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.770968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.770970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145795968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697153.770976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704d180 x1874531145795456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.770986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145795456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.770992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.770996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.770998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697153.771004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704ed80 x1874531145795584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.771014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145795584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.771020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.771023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.771026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697153.771031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704dc00 x1874531145795712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.771041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145795712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.771047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.771050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.771053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.771059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704fb80 x1874531145795840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.771068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145795840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.771074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.771077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.771080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145795840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697153.771086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697150/real 1787697153] req@ffff8f518704f800 x1874531145795328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697153.771097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145795328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697153.771103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697153.771106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697153.771109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145795328:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787697154.666521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697155.818459:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818467:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697155.818470:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818478:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818485:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818490:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697155.818516:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145796480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697155.818541:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818543:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697155.818544:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818551:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818553:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818557:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818563:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697155.818572:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145796608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697155.818583:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697155.818584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145796480:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697155.818590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697155.818594:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818598:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818602:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818606:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818611:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697155.818624:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145796736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697155.818633:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818635:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697155.818637:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818640:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818643:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818647:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818652:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697155.818662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697155.818664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145796864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697155.818675:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818678:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697155.818680:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818685:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818688:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818692:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697155.818705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f5187389f80 x1874531145796480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697155.818711:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145796992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697155.818729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697155.818733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697155.818736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145796480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697155.818740:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818747:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697155.818754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697155.818754:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697155.818763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697155.818763:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697155.818770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145796480:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697155.818780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145797120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697155.818790:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818793:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697155.818794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818798:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00100000:1.0:1787697155.818802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697155.818804:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818813:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818819:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697155.818832:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145797248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697155.818838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697155.818851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697155.818854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697155.818857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697155.818858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f5187389180 x1874531145796608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697155.818865:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818872:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818878:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818883:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697155.818884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145796608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.818896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697155.818898:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145797376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697155.818902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.818910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697155.818916:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697155.818921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697155.818922:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697155.818926:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697155.818932:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5055 00000100:00080000:3.0:1787697155.818935:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697155.818939:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697155.818944:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697155.818949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697155.818973:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145797504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697155.818975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f518738aa00 x1874531145796736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145796736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697155.819028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697155.819049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697155.819059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f5187389500 x1874531145796864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145796864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697155.819088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697155.819104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697155.819114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f518738a300 x1874531145796992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145796992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145796992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697155.819143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697155.819160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697155.819169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f518738bb80 x1874531145797120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145797120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697155.819219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697155.819270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697155.819281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f518738b800 x1874531145797248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145797248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697155.819311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697155.819328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697155.819339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f518738ad80 x1874531145797376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145797376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697155.819407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697155.819425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697155.819435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697155/real 1787697155] req@ffff8f5187389f80 x1874531145797504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697155.819445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145797504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697155.819452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697155.819455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697155.819457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145797504:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697156.714513:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697157.738501:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697158.762511:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697160.810587:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697160.938452:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938463:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697160.938467:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938478:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938490:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938497:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697160.938538:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145797632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697160.938564:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938568:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697160.938570:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938575:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938578:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938582:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938593:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697160.938599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145797632:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697160.938604:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145797760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697160.938613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938615:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697160.938617:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938620:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938623:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938626:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697160.938639:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145797888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697160.938646:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938649:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697160.938651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938654:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938660:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697160.938671:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145798016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697160.938675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697160.938680:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938683:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697160.938684:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938687:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938690:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938693:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697160.938706:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145798144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697160.938713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938717:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697160.938718:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938722:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938726:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938730:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938734:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697160.938742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183647100 x1874531145797632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697160.938743:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145798272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697160.938753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938756:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697160.938757:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938760:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00100000:3.0:1787697160.938760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145797632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697160.938764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938768:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938773:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697160.938779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697160.938782:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145798400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697160.938783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.938787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145797632:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697160.938790:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938793:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697160.938794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938797:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938800:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938803:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938807:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697160.938815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145798528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697160.938820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145797760:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697160.938825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697160.938828:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697160.938829:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697160.938833:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5060 00000100:00080000:1.0:1787697160.938835:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697160.938839:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:3.0:1787697160.938841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697160.938844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697160.938853:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145798656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697160.938853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183645180 x1874531145797760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.938866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145797760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.938873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.938876:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.938879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145797760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697160.938885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145797888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697160.938904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.938914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183646d80 x1874531145797888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.938926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145797888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.938931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.938933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.938934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145797888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697160.938938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145798016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697160.938951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.938957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183647480 x1874531145798016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.938964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145798016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.938968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.938970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.938971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145798016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697160.938975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145798144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697160.938984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.938989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183646680 x1874531145798144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.938996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145798144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.939004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.939006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.939008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145798144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697160.939010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697160.939020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.939025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183647800 x1874531145798272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.939031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145798272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.939036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.939038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.939040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697160.939043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697160.939052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.939057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183644000 x1874531145798400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.939064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145798400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.939068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.939070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.939072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697160.939086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697160.939105:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.939114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183644700 x1874531145798528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.939121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145798528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.939126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.939127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.939129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697160.939132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697160.939152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697160.939159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697160/real 1787697160] req@ffff8f5183644e00 x1874531145798656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697160.939166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145798656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.939171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697160.939173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697160.939175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145798656:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697161.834579:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697162.858512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697164.906478:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697165.930504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697166.058515:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.058525:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697166.058529:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697166.058538:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.058549:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697166.058555:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697166.058589:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145798784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697166.058618:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.058622:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697166.058623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697166.058629:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.058632:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697166.058638:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697166.058647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697166.058661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145798912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697166.058672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145798784:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697166.058680:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.058685:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697166.058686:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697166.058692:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.058696:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697166.058701:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697166.058708:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697166.058730:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145799040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697166.058740:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.058743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697166.058746:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697166.058752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697166.058758:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.058764:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697166.058770:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697166.058776:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697166.058789:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145799168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697166.058798:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.058802:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697166.058803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697166.058808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.058813:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697166.058819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f5187388000 x1874531145798784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697166.058819:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697166.058831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697166.058854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145799296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697166.058859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145798784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697166.058902:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.058911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.058916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145798784:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697166.058918:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.058927:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697166.058930:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697166.058936:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.058940:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697166.058942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145798912:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697166.058952:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697166.058963:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697166.058968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697166.058981:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145799424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697166.058983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f518738b100 x1874531145798912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697166.059035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145798912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697166.059042:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697166.059045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697166.059048:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697166.059051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145798912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697166.059077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145799040:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697166.059080:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697166.059088:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:3.0:1787697166.059094:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697166.059103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697166.059104:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697166.059117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f5187388380 x1874531145799040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697166.059121:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697166.059135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145799040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697166.059141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.059149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697166.059149:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145799552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697166.059156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145799040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697166.059175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145799168:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697166.059196:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.059199:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697166.059202:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697166.059204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697166.059219:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00100000:1.0:1787697166.059221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f518738a680 x1874531145799168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697166.059228:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697166.059242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145799168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697166.059243:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697166.059259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697166.059260:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697166.059266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145799168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697166.059278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145799296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697166.059286:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145799680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697166.059296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697166.059305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f5187388e00 x1874531145799296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697166.059308:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697166.059326:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697166.059332:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697166.059334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145799296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697166.059352:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5065 00000100:00080000:1.0:1787697166.059375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.059379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145799296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697166.059401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799424:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697166.059407:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697166.059417:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697166.059421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697166.059423:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697166.059431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f518738b480 x1874531145799424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697166.059444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145799424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697166.059447:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145799808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697166.059455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.059465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697166.059486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697166.059503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697166.059511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f5187389c00 x1874531145799552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697166.059520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145799552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697166.059529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.059531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697166.059536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697166.059545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697166.059551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f5187388a80 x1874531145799680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697166.059556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145799680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697166.059560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.059562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697166.059572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697166.059581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697166.059587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697166/real 1787697166] req@ffff8f5187388e00 x1874531145799808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697166.059593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145799808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697166.059597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697166.059599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697166.059601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145799808:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697166.954499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697169.002486:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697170.026531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697171.050531:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697171.178448:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178459:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697171.178463:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178473:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178483:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178490:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697171.178525:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145799936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697171.178548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697171.178554:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178559:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178563:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178566:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178576:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697171.178589:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145800064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697171.178607:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697171.178611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178614:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178617:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178620:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178624:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697171.178634:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145800192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697171.178643:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178646:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697171.178648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178651:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178654:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178657:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697171.178673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145800320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697171.178675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145799936:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697171.178685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178687:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697171.178688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178691:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178693:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178698:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697171.178711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145800448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697171.178717:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697171.178720:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178722:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178724:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178726:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697171.178736:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145800576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697171.178742:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178745:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697171.178747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178750:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178756:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178760:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697171.178769:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145800704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697171.178777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178780:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697171.178781:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178784:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178795:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178799:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697171.178816:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145800832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697171.178818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697171.178829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697171.178832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697171.178833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697171.178837:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5070 00000100:00080000:1.0:1787697171.178843:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697171.178846:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697171.178850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697171.178860:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145800960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697171.178940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183647b80 x1874531145799936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.178994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145799936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.179002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.179004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.179007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145799936:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697171.179017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697171.179041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.179053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183646a00 x1874531145800064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.179064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145800064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.179071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.179073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.179075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697171.179100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697171.179118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.179128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183645c00 x1874531145800192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.179138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145800192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.179145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.179148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.179151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697171.179156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697171.179172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.179182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183644380 x1874531145800320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.179191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145800320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.179198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.179200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.179203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697171.179208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697171.179224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.179233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183644e00 x1874531145800448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.179263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145800448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.179280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.179283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.179285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145800448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697171.179290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697171.179320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.179349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183644700 x1874531145800576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.179381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145800576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.179409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.179484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.179620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697171.179691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697171.179841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.179942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183644000 x1874531145800704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.180009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145800704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.180050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.180073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.180077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697171.180083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697171.180171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.180247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183647800 x1874531145800832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.180282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145800832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.180326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.180340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.180374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697171.180379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697171.180445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697171.180455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697171/real 1787697171] req@ffff8f5183646680 x1874531145800960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697171.180498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145800960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697171.180505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697171.180530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697171.180533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145800960:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697173.098492:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697174.122529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697175.146489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697176.298448:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298457:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697176.298461:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298470:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298479:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.298487:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697176.298519:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145801088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697176.298553:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298557:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697176.298559:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298563:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298567:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.298571:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298581:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697176.298594:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145801216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697176.298605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145801088:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697176.298608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298613:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697176.298614:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298622:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.298626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697176.298643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145801344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697176.298653:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697176.298659:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298662:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298666:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.298669:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697176.298682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697176.298687:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145801472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697176.298698:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298702:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697176.298704:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298709:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298712:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.298715:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697176.298731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145801600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697176.298741:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298746:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787697176.298746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183647480 x1874531145801088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697176.298749:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298758:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298763:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697176.298767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145801088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697176.298771:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697176.298782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.298788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.298791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145801088:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697176.298803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145801728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697176.298809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697176.298824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697176.298828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697176.298830:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697176.298831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697176.298839:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298844:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697176.298844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183646d80 x1874531145801216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697176.298852:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298861:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697176.298869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145801216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697176.298888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145801856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697176.298915:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697176.298919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697176.298920:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697176.298924:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298929:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298932:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.298935:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.298940:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787697176.298944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697176.298954:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145801984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697176.298966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697176.298968:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697176.298974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801344:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697176.298984:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697176.298986:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697176.298991:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5075 00000100:00080000:1.0:1787697176.298996:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697176.299000:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697176.299004:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697176.299012:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145802112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697176.299019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183645180 x1874531145801344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145801344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697176.299087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697176.299109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183647100 x1874531145801472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145801472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697176.299152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697176.299171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183646680 x1874531145801600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145801600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145801600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697176.299228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145801728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697176.299240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183647800 x1874531145801728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145801728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145801728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697176.299297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145801856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697176.299308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183644000 x1874531145801856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145801856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145801856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697176.299351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145801984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697176.299455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183644700 x1874531145801984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145801984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145801984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697176.299494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145802112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697176.299518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697176.299535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697176/real 1787697176] req@ffff8f5183644e00 x1874531145802112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697176.299561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145802112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697176.299569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697176.299586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697176.299589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145802112:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697177.194484:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697178.218536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697179.242552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697181.290750:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697181.418997:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419007:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697181.419011:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419020:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419031:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419037:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697181.419071:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145802240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419104:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419108:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697181.419110:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419115:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419118:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419122:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419130:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697181.419141:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145802368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419151:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419154:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697181.419155:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419159:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419162:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419165:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419169:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697181.419178:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145802496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419186:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419189:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697181.419191:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419194:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419197:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419200:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419204:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697181.419213:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145802624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419221:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419225:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697181.419226:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419229:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419232:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419235:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419239:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697181.419249:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145802752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419264:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419267:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697181.419269:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419272:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419275:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419279:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419282:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697181.419291:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145802880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419299:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419302:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697181.419304:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419307:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419309:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419312:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419316:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697181.419323:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145803008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419331:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419334:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697181.419336:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419340:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419343:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419348:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419380:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697181.419391:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145803136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697181.419401:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697181.419404:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697181.419406:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697181.419410:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5081 00000100:00080000:1.0:1787697181.419426:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697181.419429:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697181.419433:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697181.419445:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145803264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697182.314543:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697183.338544:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697185.386555:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697186.410520:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697186.538610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538620:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697186.538624:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538633:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538642:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538648:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697186.538682:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145803392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538717:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538721:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697186.538722:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538725:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538728:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538731:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697186.538749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145803520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538757:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538760:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697186.538763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538769:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538771:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538773:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538776:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697186.538784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145803648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697186.538793:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538795:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538797:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538800:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538803:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697186.538808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145803776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697186.538830:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538839:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538844:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538848:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697186.538862:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145803904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538873:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697186.538882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538889:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538892:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538897:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538904:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697186.538916:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145804032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538928:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538932:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697186.538934:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538941:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538944:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538946:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697186.538954:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145804160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.538962:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.538970:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697186.538971:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.538976:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.538980:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.538984:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.538988:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697186.539006:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145804288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697186.539036:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697186.539041:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697186.539042:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697186.539050:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5086 00000100:00080000:3.0:1787697186.539055:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697186.539060:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697186.539064:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697186.539077:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145804416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697187.434574:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697189.482558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697190.506556:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697191.530587:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697191.658488:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658498:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697191.658503:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658513:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658524:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658530:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697191.658576:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145804544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658623:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697191.658625:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658629:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658633:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658637:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658648:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697191.658663:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145804672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658677:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697191.658682:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658685:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658688:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658691:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697191.658708:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145804800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658717:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697191.658721:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658724:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658731:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658734:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658739:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697191.658750:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145804928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658768:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697191.658770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658778:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658782:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697191.658802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145805056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658819:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697191.658831:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658837:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658844:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658851:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658859:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697191.658869:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145805184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658888:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658891:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697191.658893:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658897:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658900:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658905:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658910:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697191.658921:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145805312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658933:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697191.658941:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658945:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.658947:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.658953:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.658959:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697191.658968:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145805440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697191.658984:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697191.658988:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697191.658989:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697191.658992:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5091 00000100:00080000:1.0:1787697191.659002:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697191.659006:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697191.659011:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697191.659025:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145805568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787697193.578612:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697194.602582:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697195.626606:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697196.778426:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778434:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697196.778438:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778447:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778456:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778463:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697196.778496:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145805696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778520:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778533:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697196.778535:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778540:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778544:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778548:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778557:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697196.778572:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145805824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778587:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697196.778589:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778593:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697196.778615:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145805952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778626:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778630:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697196.778632:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778638:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778642:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778646:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778651:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697196.778662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145806080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697196.778693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778701:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778705:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778709:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697196.778730:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145806208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778747:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778753:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697196.778757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778766:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778772:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778779:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778786:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697196.778800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145806336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778820:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778825:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697196.778828:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778838:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778843:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778851:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697196.778878:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145806464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778888:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778892:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697196.778895:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778902:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778909:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778914:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778921:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697196.778936:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145806592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697196.778949:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697196.778956:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697196.778960:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697196.778965:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5096 00000100:00080000:1.0:1787697196.778969:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697196.778973:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697196.778977:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697196.778985:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145806720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787697197.674611:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697198.698465:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697199.722515:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697201.770540:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697201.898482:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898492:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697201.898496:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898507:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898518:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898524:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697201.898564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145806848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898590:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697201.898594:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898598:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898600:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898604:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898613:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697201.898627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145806976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898638:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898641:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697201.898643:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898649:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898652:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898658:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898663:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697201.898671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145807104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697201.898682:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898685:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898687:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898690:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697201.898699:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145807232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697201.898710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898713:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898717:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898725:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697201.898744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145807360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697201.898758:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898762:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898766:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898773:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697201.898795:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145807488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898806:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898811:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697201.898813:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898819:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898821:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898826:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898830:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697201.898838:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145807616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898844:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697201.898851:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898855:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898857:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898862:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898867:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697201.898880:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145807744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697201.898893:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697201.898898:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697201.898901:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697201.898909:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5101 00000100:00080000:1.0:1787697201.898912:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697201.898914:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697201.898916:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697201.898923:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145807872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697202.794578:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697203.818495:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697205.866604:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697206.890598:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697207.018480:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018491:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697207.018496:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018506:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018518:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018525:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697207.018561:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145808000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697207.018591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018596:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018604:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018616:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697207.018632:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145808128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018645:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018648:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697207.018650:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018654:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018658:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018662:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018667:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697207.018680:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145808256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018696:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697207.018703:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018708:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018712:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018716:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018722:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697207.018738:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145808384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018752:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018760:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697207.018763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018769:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018777:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018785:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018791:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697207.018802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145808512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697207.018812:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018817:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018820:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018825:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697207.018857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145808640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018875:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018881:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697207.018886:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018895:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018902:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018912:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018919:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697207.018936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145808768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.018944:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.018949:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697207.018951:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.018957:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.018961:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.018967:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.018974:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697207.018986:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145808896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697207.019003:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697207.019012:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697207.019017:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697207.019028:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5106 00000100:00080000:3.0:1787697207.019035:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697207.019043:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697207.019051:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697207.019071:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145809024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697207.914595:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697209.962531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697210.986545:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697212.010514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697212.138511:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138523:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697212.138528:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138538:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138550:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138558:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697212.138605:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145809152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138634:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138644:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697212.138647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138652:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138657:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138662:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697212.138687:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145809280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138718:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138722:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697212.138724:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138730:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138734:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138739:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697212.138759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145809408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697212.138779:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138786:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138790:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138793:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138798:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697212.138808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145809536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138816:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697212.138820:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138825:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138832:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138838:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138843:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697212.138855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145809664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138879:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138885:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697212.138888:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138896:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138900:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138907:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138910:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697212.138917:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145809792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138923:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138926:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697212.138927:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138929:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138932:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138937:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138940:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697212.138950:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145809920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138962:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.138965:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697212.138966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.138969:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.138971:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.138977:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.138982:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697212.138990:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145810048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697212.138998:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697212.139004:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697212.139007:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697212.139014:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5111 00000100:00080000:1.0:1787697212.139018:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697212.139023:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697212.139026:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697212.139035:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145810176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787697214.058545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697215.082563:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697216.106501:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697217.258497:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258509:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697217.258514:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258526:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258540:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258548:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697217.258597:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145810304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258641:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697217.258647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258652:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258656:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258660:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697217.258684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145810432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258698:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697217.258700:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258704:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258707:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258711:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258716:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697217.258728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145810560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697217.258744:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258748:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258752:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258756:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258761:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697217.258772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145810688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258781:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697217.258787:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258791:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258796:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258799:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697217.258816:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145810816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697217.258830:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258834:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258839:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258845:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258851:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697217.258867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145810944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258881:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697217.258883:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258887:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258891:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258895:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258901:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697217.258916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145811072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258928:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258932:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697217.258934:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.258938:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.258944:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.258952:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.258960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697217.258976:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145811200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697217.258986:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697217.258996:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697217.258998:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697217.259003:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5116 00000100:00080000:1.0:1787697217.259009:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697217.259014:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697217.259020:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697217.259039:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145811328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697218.154560:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697219.178541:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697220.202566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787697220.330626:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787697220.330669:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787697220.330684:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330735:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330746:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330755:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330763:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330770:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330777:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330784:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697220.330792:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00010000:00010000:3.0:1787697222.250502:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697222.378443:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378451:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697222.378454:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378462:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378471:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378478:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697222.378505:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145811456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378528:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378531:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697222.378532:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378535:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378538:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378540:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378547:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697222.378561:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145811584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378574:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378576:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697222.378577:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378581:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378588:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378596:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697222.378607:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145811712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378621:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697222.378624:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378629:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378649:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697222.378671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145811840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378691:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697222.378694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378705:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378710:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697222.378731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145811968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378759:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378763:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697222.378767:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378777:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378785:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378793:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697222.378823:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145812096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378842:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697222.378855:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378861:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378867:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378871:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378876:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697222.378893:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145812224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378916:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.378925:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697222.378930:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.378939:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.378947:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.378956:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.378967:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697222.378985:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145812352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697222.378996:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697222.379000:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697222.379002:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697222.379010:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5122 00000100:00080000:1.0:1787697222.379016:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697222.379021:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697222.379027:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697222.379043:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145812480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697223.274517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697224.298493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697226.347378:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697227.370559:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697227.498478:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498489:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697227.498493:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498503:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498514:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498520:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697227.498555:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145812608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498582:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697227.498588:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498601:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498605:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498614:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697227.498627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145812736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498639:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697227.498647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498652:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498655:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498658:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697227.498671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145812864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498681:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697227.498686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498689:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498692:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498696:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697227.498710:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145812992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498720:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697227.498730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498734:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498738:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498742:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697227.498759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145813120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498769:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697227.498778:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498787:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498792:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498797:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498803:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697227.498817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145813248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498833:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697227.498840:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498847:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498850:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498853:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498857:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697227.498884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145813376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498895:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498901:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697227.498905:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498911:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.498915:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.498921:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.498929:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697227.498947:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145813504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697227.498965:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697227.498979:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697227.498982:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697227.498990:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5127 00000100:00080000:1.0:1787697227.499006:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697227.499010:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697227.499015:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697227.499030:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145813632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697228.394501:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697230.442514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697231.466522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697232.490551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697232.618467:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618476:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697232.618479:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618488:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618496:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618501:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697232.618530:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145813760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697232.618552:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618555:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618562:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618570:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697232.618582:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145813888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618592:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618595:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697232.618597:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618601:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618607:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618615:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697232.618634:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145814016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697232.618651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618653:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618656:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618658:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618661:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697232.618671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145814144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697232.618681:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618683:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618685:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618687:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697232.618697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145814272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618713:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697232.618715:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618720:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618724:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618730:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697232.618745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145814400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618766:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697232.618769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618776:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618778:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618780:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618783:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697232.618790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145814528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697232.618807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618812:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618816:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618821:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697232.618837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145814656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697232.618880:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697232.618882:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697232.618884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697232.618887:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5132 00000100:00080000:2.0:1787697232.618889:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697232.618892:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697232.618894:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697232.618915:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145814784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697234.538514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697235.562514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697236.586540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697237.738447:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697237.738457:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738465:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738473:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738479:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697237.738506:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145814912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738529:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738533:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697237.738534:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738538:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738542:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738546:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738556:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697237.738569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145815040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738587:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738589:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697237.738590:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738594:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697237.738617:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145815168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738625:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738628:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697237.738629:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738633:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738647:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738655:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738660:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697237.738670:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145815296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738677:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697237.738682:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738688:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738695:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738700:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738705:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697237.738721:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145815424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738733:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738738:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697237.738740:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738747:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738749:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738752:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697237.738764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145815552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738779:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738781:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697237.738782:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738785:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738790:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738794:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697237.738808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145815680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738843:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697237.738847:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738852:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738858:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738873:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697237.738888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145815808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697237.738899:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697237.738902:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697237.738904:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697237.738907:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5137 00000100:00080000:1.0:1787697237.738913:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697237.738920:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697237.738924:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697237.738937:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145815936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697238.634515:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697239.658517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697240.682502:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697242.730581:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697242.858477:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697242.858491:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858517:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858528:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858535:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697242.858569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145816064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858595:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858598:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697242.858600:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858604:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858608:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858611:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.858620:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697242.858633:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145816192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858642:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697242.858646:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858650:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858653:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858660:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.858668:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697242.858680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145816320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858689:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697242.858694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858697:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858701:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858713:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.858719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697242.858759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145816448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858771:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697242.858775:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858781:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858787:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858791:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.858796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697242.858810:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145816576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858843:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697242.858852:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858862:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858866:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858870:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.858876:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697242.858891:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145816704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858918:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858925:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697242.858929:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.858937:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.858945:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.858953:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.858960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697242.858976:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145816832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.858988:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.858994:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697242.858997:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.859002:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.859007:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.859015:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.859020:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697242.859031:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145816960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697242.859048:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697242.859058:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697242.859063:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697242.859071:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5142 00000100:00080000:2.0:1787697242.859075:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697242.859082:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697242.859088:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697242.859102:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145817088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697243.754511:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697244.778489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697246.826625:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697247.850613:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697247.978693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145817216:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697247.978780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.978833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace90380 x1874531145817216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.978855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145817216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.978893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.978905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.978916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145817216:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697247.978947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697247.978996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace93800 x1874531145817344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145817344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697247.979112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697247.979133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace90000 x1874531145817472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145817472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697247.979181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697247.979202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace92680 x1874531145817600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145817600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697247.979240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697247.979253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace90e00 x1874531145817728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145817728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145817728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697247.979285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145817856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697247.979313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace93100 x1874531145817856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145817856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145817856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697247.979425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145817984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697247.979441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace93b80 x1874531145817984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145817984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145817984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697247.979472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145818112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697247.979481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace90380 x1874531145818112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145818112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145818112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697247.979528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145818240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697247.979544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697247.979551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697247/real 1787697247] req@ffff8f51ace90700 x1874531145818240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697247.979558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145818240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.979563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697247.979565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697247.979567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145818240:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697248.874631:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697250.922580:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697251.946613:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697252.970514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697253.098514:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098525:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697253.098529:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.098540:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.098565:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697253.098575:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697253.098631:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145818368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697253.098660:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098664:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697253.098666:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.098672:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.098676:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697253.098681:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697253.098689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145818368:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787697253.098693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697253.098710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145818496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697253.098721:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098725:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697253.098727:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.098731:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.098735:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697253.098740:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697253.098745:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697253.098757:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145818624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697253.098774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697253.098775:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098781:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697253.098783:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.098788:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.098792:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697253.098797:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697253.098803:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697253.098814:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145818752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697253.098823:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697253.098829:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.098835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00100000:3.0:1787697253.098837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704fb80 x1874531145818368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697253.098840:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697253.098848:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697253.098855:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697253.098864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145818368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697253.098869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145818880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697253.098886:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098893:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697253.098896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697253.098898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697253.098902:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.098908:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697253.098909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697253.098915:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697253.098918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145818368:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787697253.098922:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697253.098936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145819008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697253.098945:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.098950:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787697253.098951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818496:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697253.098953:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.098960:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.098965:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697253.098969:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697253.098975:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697253.098976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697253.098990:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145819136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697253.098996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704f800 x1874531145818496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697253.099007:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.099012:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697253.099016:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697253.099023:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.099028:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697253.099032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145818496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697253.099036:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697253.099044:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787697253.099048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697253.099058:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145819264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697253.099061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697253.099074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697253.099088:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697253.099092:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697253.099094:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697253.099100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697253.099101:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5152 00000100:00080000:2.0:1787697253.099108:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697253.099113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704d880 x1874531145818624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697253.099115:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697253.099123:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697253.099127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145818624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697253.099140:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145819392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697253.099143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697253.099153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697253.099171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697253.099180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704c700 x1874531145818752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697253.099188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145818752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697253.099202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697253.099214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697253.099236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697253.099247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704c380 x1874531145818880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697253.099259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145818880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697253.099274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145818880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697253.099279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697253.099296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697253.099305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704c000 x1874531145819008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697253.099315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145819008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697253.099330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697253.099347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697253.099392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697253.099404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704e680 x1874531145819136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697253.099415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145819136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697253.099433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697253.099439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697253.099458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697253.099467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704ce00 x1874531145819264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697253.099477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145819264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697253.099487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697253.099490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697253.099502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697253.099508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697253/real 1787697253] req@ffff8f518704df80 x1874531145819392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697253.099514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145819392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697253.099518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697253.099520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697253.099522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145819392:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697255.018514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697256.042536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697257.066516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697258.218431:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218438:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697258.218441:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218449:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218457:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218475:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697258.218512:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145819520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218538:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218543:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697258.218545:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218550:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218553:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218557:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218565:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697258.218576:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145819648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218588:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697258.218589:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218600:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697258.218612:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145819776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697258.218621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218623:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218625:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218627:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697258.218635:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145819904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218641:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697258.218644:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218646:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218648:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218653:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697258.218664:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145820032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218673:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697258.218675:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218677:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218679:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218681:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697258.218690:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145820160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218695:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218699:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697258.218701:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218708:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218713:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218719:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218722:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697258.218731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145820288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697258.218767:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218773:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218779:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218784:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218789:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697258.218797:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145820416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697258.218803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697258.218805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697258.218805:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697258.218808:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5157 00000100:00080000:3.0:1787697258.218810:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697258.218814:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697258.218817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697258.218826:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145820544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697259.114571:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697260.138566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697261.162551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697263.210589:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697263.338422:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338433:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697263.338437:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338448:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338459:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338465:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697263.338497:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145820672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338523:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338527:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697263.338528:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338532:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338535:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338539:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338549:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697263.338562:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145820800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338574:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338577:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697263.338579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338583:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338586:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338590:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697263.338613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145820928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338639:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697263.338645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338650:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338655:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338659:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338664:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697263.338680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145821056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338689:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697263.338693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338697:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338701:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338705:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697263.338720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145821184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697263.338744:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338749:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338756:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338761:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697263.338773:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145821312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697263.338815:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338820:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338825:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338828:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338833:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697263.338851:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145821440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338878:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338891:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697263.338893:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338897:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338902:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338908:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338916:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697263.338938:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145821568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697263.338949:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697263.338953:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697263.338959:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697263.338972:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5162 00000100:00080000:3.0:1787697263.338977:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697263.338985:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697263.338995:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697263.339020:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145821696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697264.234503:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697265.258498:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697267.306492:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697268.330549:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697268.458591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145821824:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697268.458650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.458685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace91880 x1874531145821824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.458712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145821824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.458727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.458733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.458737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145821824:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697268.458746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145821952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697268.458768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.458783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace91180 x1874531145821952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.458812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145821952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.458840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.458847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.458851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145821952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697268.458875:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145822080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697268.458898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.458912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace92d80 x1874531145822080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.458920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145822080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.458925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.458927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.458930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145822080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697268.458934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145822208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697268.458942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.458947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace93480 x1874531145822208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.458953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145822208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.458957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.458959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.458961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145822208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697268.458964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145822336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697268.458973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.458978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace92300 x1874531145822336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.458984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145822336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.458988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.458989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.458991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145822336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697268.458994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697268.459003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.459008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace91500 x1874531145822464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.459014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145822464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.459019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.459020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.459021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697268.459024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697268.459032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.459038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace91c00 x1874531145822592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.459044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145822592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.459048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.459050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.459051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697268.459054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697268.459061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.459066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace90700 x1874531145822720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.459071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145822720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.459075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.459077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.459078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697268.459088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697268.459096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697268.459101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697268/real 1787697268] req@ffff8f51ace90380 x1874531145822848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697268.459107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145822848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697268.459112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697268.459113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697268.459115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145822848:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697269.354558:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697271.402541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697272.426518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697273.450506:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697273.578471:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578478:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697273.578481:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578489:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578497:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578502:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697273.578533:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145822976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578560:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578564:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697273.578566:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578570:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578573:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578575:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578583:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697273.578593:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145823104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578604:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578607:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697273.578609:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578612:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578615:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578618:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578622:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697273.578633:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145823232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578642:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697273.578648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578654:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578660:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578664:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578668:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697273.578677:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145823360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697273.578691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578698:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578704:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578707:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578713:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697273.578731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145823488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578747:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578750:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697273.578752:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578758:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578763:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578766:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578770:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697273.578782:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145823616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578793:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697273.578801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578807:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578820:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697273.578830:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145823744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697273.578852:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578856:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578860:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578864:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578867:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697273.578880:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145823872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697273.578907:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697273.578913:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697273.578916:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697273.578922:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5173 00000100:00080000:3.0:1787697273.578927:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697273.578929:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697273.578931:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697273.578939:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145824000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697275.498520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697276.522493:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697277.546530:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697278.698465:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698475:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697278.698479:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698489:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698499:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698505:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697278.698537:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145824128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698565:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697278.698570:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698575:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698578:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698581:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698590:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697278.698603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145824256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697278.698618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698631:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698635:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698640:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698645:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697278.698658:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145824384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698683:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697278.698685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698690:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698694:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698698:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697278.698714:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145824512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698722:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697278.698729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698737:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698742:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698747:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697278.698768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145824640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697278.698789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698797:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698801:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698805:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698809:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697278.698822:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145824768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698833:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697278.698838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698843:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698849:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698855:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697278.698878:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145824896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698905:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698911:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697278.698914:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698922:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698927:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698934:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698941:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697278.698953:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145825024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697278.698969:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697278.698974:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697278.698976:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697278.698981:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5178 00000100:00080000:3.0:1787697278.698984:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697278.698990:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697278.698996:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697278.699010:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145825152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697279.594531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697280.618569:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697281.642564:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697283.690507:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697283.818460:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818470:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697283.818474:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818483:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818501:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697283.818535:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145825280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818559:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818561:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697283.818563:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818566:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818569:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818571:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818578:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697283.818586:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145825408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818593:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818595:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697283.818596:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818598:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818600:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818606:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697283.818620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145825536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697283.818636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818639:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818643:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818647:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818652:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697283.818664:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145825664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818674:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818677:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697283.818678:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818688:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818692:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818695:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697283.818711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145825792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697283.818730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818736:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818741:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818745:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697283.818763:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145825920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818783:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697283.818784:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818789:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818792:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818797:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697283.818816:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145826048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818823:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818830:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697283.818832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818836:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818840:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818844:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818849:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697283.818867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145826176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697283.818877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697283.818880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697283.818881:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697283.818883:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5183 00000100:00080000:3.0:1787697283.818886:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697283.818891:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697283.818898:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697283.818913:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145826304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697284.714522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697285.738527:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697286.890604:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787697286.890640:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697286.890656:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890706:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890719:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890729:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890740:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890752:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890763:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890775:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697286.890794:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00010000:00010000:3.0:1787697287.786542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697288.810569:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697288.938493:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938504:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697288.938508:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938522:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938536:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938545:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697288.938588:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145826432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.938625:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938629:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697288.938631:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938636:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938640:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938643:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.938653:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697288.938669:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145826560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.938680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697288.938688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938694:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938700:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938704:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.938709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697288.938723:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145826688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.938734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697288.938739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938744:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938747:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938750:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.938755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697288.938765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145826816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.938774:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938778:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697288.938780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938784:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938804:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938810:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.938816:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697288.938832:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145826944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.938842:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938846:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697288.938848:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938854:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938858:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938864:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.938871:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697288.938888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145827072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.938898:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.938903:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697288.938906:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.938916:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.938924:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.938966:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.938973:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697288.938986:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145827200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.939004:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.939008:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697288.939010:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.939016:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.939021:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.939027:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.939036:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697288.939047:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145827328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697288.939062:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697288.939069:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697288.939071:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697288.939076:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5188 00000100:00080000:3.0:1787697288.939083:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697288.939090:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697288.939097:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697288.939112:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145827456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697289.834526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697291.882573:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697292.906625:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697293.930529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697294.058459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697294.058471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058481:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058490:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058497:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697294.058541:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145827584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058571:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058575:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697294.058577:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058586:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058590:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058599:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697294.058616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145827712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697294.058636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058640:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058643:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058647:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058651:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697294.058662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145827840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058670:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058672:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697294.058673:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058693:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058697:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058700:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058704:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697294.058714:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145827968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058724:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697294.058731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058736:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058739:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058744:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058748:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697294.058759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145828096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697294.058777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058809:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058817:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058825:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058831:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697294.058855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145828224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058873:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058877:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697294.058878:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058883:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058887:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058892:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058899:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697294.058912:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145828352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.058933:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.058938:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697294.058941:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.058963:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.058968:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.058971:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.058976:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697294.058991:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145828480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697294.059020:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697294.059028:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697294.059031:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697294.059038:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5193 00000100:00080000:3.0:1787697294.059047:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697294.059055:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697294.059061:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697294.059078:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145828608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697295.978469:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697297.002566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697298.026635:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697299.178438:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178445:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697299.178449:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178456:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178491:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178497:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697299.178529:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145828736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178551:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178554:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697299.178555:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178568:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178571:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178575:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178588:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697299.178598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145828864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178613:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697299.178615:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178627:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697299.178644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145828992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178653:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178656:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697299.178657:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178662:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178669:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697299.178681:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145829120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697299.178690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178695:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178697:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697299.178710:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145829248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178728:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697299.178732:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178735:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178737:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178742:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697299.178776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145829376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697299.178794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178797:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178799:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178802:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178805:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697299.178811:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145829504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178816:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178819:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697299.178821:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178829:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178834:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178837:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178843:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697299.178856:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145829632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697299.178877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697299.178880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697299.178881:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697299.178886:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5198 00000100:00080000:1.0:1787697299.178891:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697299.178896:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697299.178903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697299.178914:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145829760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697300.074518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697301.098503:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697302.122520:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697304.170505:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697304.298453:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298463:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697304.298467:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298476:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298490:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298497:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697304.298530:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145829888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298553:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298556:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697304.298558:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298565:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298571:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298580:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697304.298592:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145830016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298604:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298612:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697304.298615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298619:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298622:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298625:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298629:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697304.298640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145830144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298648:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298651:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697304.298652:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298659:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298662:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697304.298694:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145830272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298704:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298708:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697304.298710:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298714:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298718:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298722:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298728:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697304.298742:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145830400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298754:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298761:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697304.298764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298770:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298774:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298781:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697304.298803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145830528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298814:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697304.298824:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298831:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298834:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298838:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697304.298851:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145830656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298860:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697304.298866:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298874:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298880:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298886:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298892:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697304.298907:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145830784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697304.298925:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697304.298928:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697304.298931:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697304.298939:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5203 00000100:00080000:2.0:1787697304.298945:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697304.298948:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697304.298952:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697304.298966:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145830912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697305.194522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697306.218591:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697308.266535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697309.290515:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697309.418626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145831040:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697309.418696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.418742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace93b80 x1874531145831040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.418770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145831040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.418790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.418799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.418807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145831040:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697309.418838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697309.418871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.418892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace93100 x1874531145831168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.418915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145831168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.418931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.418937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.418944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697309.418955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697309.418969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.418977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace90e00 x1874531145831296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.418984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145831296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.418988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.418990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.418992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697309.418995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697309.419005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.419010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace92680 x1874531145831424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.419020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145831424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.419024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.419025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.419027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697309.419030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697309.419038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.419043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace90000 x1874531145831552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.419049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145831552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.419054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.419055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.419057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145831552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697309.419060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145831680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697309.419068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.419073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace93800 x1874531145831680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.419079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145831680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.419090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.419093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.419096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145831680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697309.419113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145831808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697309.419132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.419142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace93b80 x1874531145831808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.419153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145831808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.419160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.419163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.419166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145831808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697309.419171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145831936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697309.419189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.419199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace90380 x1874531145831936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.419210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145831936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.419217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.419219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.419223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145831936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697309.419228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145832064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697309.419245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697309.419254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697309/real 1787697309] req@ffff8f51ace90700 x1874531145832064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697309.419264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145832064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697309.419270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697309.419273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697309.419276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145832064:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697310.314488:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697312.362545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697313.386570:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697314.410540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697314.538648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145832192:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697314.538712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.538792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace91c00 x1874531145832192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.538810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145832192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.538828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.538844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.538855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145832192:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697314.538893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697314.538924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.538952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace91500 x1874531145832320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.538969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145832320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.538993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.538998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697314.539009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697314.539035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace92300 x1874531145832448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145832448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697314.539062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697314.539074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace93480 x1874531145832576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145832576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697314.539101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697314.539116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace92d80 x1874531145832704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145832704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145832704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697314.539156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145832832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697314.539176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace91180 x1874531145832832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145832832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145832832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697314.539213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145832960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697314.539225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace91880 x1874531145832960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145832960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145832960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697314.539258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145833088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697314.539268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace91c00 x1874531145833088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145833088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145833088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697314.539290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145833216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697314.539298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697314.539303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697314/real 1787697314] req@ffff8f51ace90700 x1874531145833216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697314.539309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145833216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697314.539313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697314.539315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697314.539316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145833216:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697316.458550:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697317.482559:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697318.506522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697319.658495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658503:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697319.658506:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658514:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658522:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658527:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697319.658556:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145833344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658575:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658579:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697319.658581:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658585:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658589:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658594:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658602:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697319.658616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145833344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697319.658619:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145833472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658632:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658636:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697319.658638:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658640:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658643:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658646:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658650:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697319.658659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145833600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658669:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697319.658672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697319.658677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697319.658679:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658683:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658686:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697319.658697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145833728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658705:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697319.658706:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658709:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658711:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697319.658714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704dc00 x1874531145833344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697319.658715:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658720:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697319.658729:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145833856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658736:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697319.658737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145833344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697319.658739:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697319.658742:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658746:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658751:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658755:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658758:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697319.658759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697319.658764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145833984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697319.658767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697319.658775:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697319.658776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145833344:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697319.658778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697319.658780:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658785:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00080000:2.0:1787697319.658789:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658791:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697319.658799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145834112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658804:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697319.658808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658811:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5219 00000100:00100000:3.0:1787697319.658812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833472:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697319.658813:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658818:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658823:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697319.658829:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145834240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697319.658839:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697319.658842:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697319.658843:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697319.658846:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5219 00000800:00000100:3.0:1787697319.658849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697319.658850:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697319.658855:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697319.658858:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697319.658864:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145834368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697319.658870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704ed80 x1874531145833472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.658885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145833472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.658894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.658898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.658902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697319.658908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697319.658928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.658939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704ea00 x1874531145833600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.658950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145833600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.658957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.658960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.658963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697319.658969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697319.658989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.659000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704e300 x1874531145833728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.659012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145833728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.659019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.659022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.659026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697319.659031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697319.659049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.659060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704f480 x1874531145833856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.659071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145833856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.659079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.659082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.659085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145833856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697319.659090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145833984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697319.659110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.659121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704d500 x1874531145833984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.659132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145833984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.659145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.659149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.659152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145833984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697319.659158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145834112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697319.659178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.659188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704d180 x1874531145834112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.659199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145834112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.659205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.659207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.659209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145834112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697319.659223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145834240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697319.659236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.659242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704dc00 x1874531145834240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.659249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145834240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.659253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.659255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.659257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145834240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697319.659268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145834368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697319.659277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697319.659283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697319/real 1787697319] req@ffff8f518704fb80 x1874531145834368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697319.659289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145834368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697319.659293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697319.659295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697319.659296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145834368:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697320.554575:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697321.578547:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697322.602501:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697324.650501:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697324.778588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145834496:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697324.778641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.778680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace90380 x1874531145834496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.778704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145834496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.778722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.778740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.778750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145834496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697324.778763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145834624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697324.778788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.778801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace93b80 x1874531145834624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.778821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145834624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.778828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.778831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.778836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145834624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697324.778860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145834752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697324.778889:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.778903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace93800 x1874531145834752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.778915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145834752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.778923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.778926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.778929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145834752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697324.778935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145834880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697324.778955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.778965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace90000 x1874531145834880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.778973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145834880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.778978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.778980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.778982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145834880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697324.778985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145835008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697324.778996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.779002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace92680 x1874531145835008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.779008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145835008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.779013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.779014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.779016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145835008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697324.779019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697324.779028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.779033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace90e00 x1874531145835136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.779039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145835136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.779048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.779050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.779052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697324.779054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697324.779063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.779068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace93100 x1874531145835264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.779073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145835264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.779078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.779079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.779081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697324.779084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697324.779091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.779096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace90700 x1874531145835392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.779102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145835392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.779106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.779107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.779110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697324.779121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697324.779130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697324.779136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697324/real 1787697324] req@ffff8f51ace91c00 x1874531145835520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697324.779142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145835520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697324.779146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697324.779148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697324.779149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145835520:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697325.674521:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697326.698521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697328.746512:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697329.770515:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697329.898603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145835648:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697329.898682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.898729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace91880 x1874531145835648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.898756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145835648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.898779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.898786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.898794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145835648:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697329.898827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145835776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697329.898869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.898883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace91180 x1874531145835776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.898902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145835776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.898909:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.898912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.898915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145835776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697329.898920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145835904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697329.898938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.898944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace92d80 x1874531145835904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.898951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145835904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.898956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.898958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.898960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145835904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697329.898963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145836032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697329.898973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.898978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace93480 x1874531145836032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.898989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145836032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.898997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.898999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.899002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145836032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697329.899008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145836160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697329.899025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.899035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace92300 x1874531145836160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.899046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145836160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.899052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.899054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.899056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145836160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697329.899060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697329.899073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.899079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace91500 x1874531145836288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.899086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145836288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.899095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.899097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.899099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697329.899110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697329.899119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.899124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace91880 x1874531145836416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.899130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145836416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.899135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.899137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.899139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697329.899142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697329.899150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.899155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace91c00 x1874531145836544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.899161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145836544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.899165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.899167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.899169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697329.899172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697329.899180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697329.899185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697329/real 1787697329] req@ffff8f51ace90700 x1874531145836672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697329.899191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145836672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697329.899195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697329.899197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697329.899198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145836672:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697330.794539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697332.842488:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697333.866524:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697334.890586:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697335.018478:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697335.018490:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018498:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00080000:1.0:1787697335.018507:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697335.018516:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697335.018548:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145836800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697335.018571:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018574:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697335.018576:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018581:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00080000:1.0:1787697335.018584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697335.018588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145836800:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697335.018599:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018608:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697335.018624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145836928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697335.018632:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697335.018636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018638:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5234 00000800:00000100:3.0:1787697335.018646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697335.018648:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697335.018653:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697335.018665:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145837056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697335.018671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018674:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697335.018675:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018677:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00100000:3.0:1787697335.018681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183646a00 x1874531145836800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697335.018684:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697335.018690:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018695:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697335.018699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145836800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697335.018707:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145837184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697335.018710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.018716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.018723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145836800:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697335.018725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697335.018731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018736:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00080000:1.0:1787697335.018740:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697335.018741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145836928:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697335.018746:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697335.018759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697335.018766:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145837312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697335.018771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183645c00 x1874531145836928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697335.018776:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018782:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697335.018785:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018791:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00080000:1.0:1787697335.018795:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697335.018799:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697335.018800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145836928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697335.018803:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697335.018812:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145837440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697335.018814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.018824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697335.018828:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697335.018833:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787697335.018833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145836928:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697335.018836:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00080000:1.0:1787697335.018844:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697335.018846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145837056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697335.018849:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018854:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697335.018862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145837568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697335.018867:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697335.018871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697335.018873:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697335.018876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018879:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00080000:1.0:1787697335.018882:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697335.018886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183644380 x1874531145837056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697335.018887:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018893:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697335.018903:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145837056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697335.018904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145837696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697335.018922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697335.018925:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697335.018927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697335.018930:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787697335.018933:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145837056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697335.018934:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697335.018942:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5234 00000100:00100000:3.0:1787697335.018944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145837184:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697335.018946:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697335.018952:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697335.018955:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697335.018966:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145837824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697335.018966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697335.018978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183647b80 x1874531145837184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697335.018988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145837184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697335.018993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.018995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.018997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145837184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697335.019006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145837312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697335.019016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697335.019021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183646a00 x1874531145837312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697335.019027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145837312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697335.019032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.019034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.019035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145837312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697335.019038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697335.019046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697335.019050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183646d80 x1874531145837440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697335.019056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145837440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697335.019065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.019067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.019069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697335.019071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697335.019079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697335.019084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183645180 x1874531145837568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697335.019090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145837568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697335.019095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.019098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.019101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697335.019106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697335.019122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697335.019132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183647800 x1874531145837696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697335.019156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145837696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697335.019163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.019165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.019167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697335.019171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697335.019184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697335.019191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697335/real 1787697335] req@ffff8f5183647100 x1874531145837824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697335.019197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145837824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697335.019205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697335.019207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697335.019210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145837824:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697336.938539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697337.962504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697338.986519:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697340.138476:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697340.138487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697340.138491:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138501:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138511:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138519:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697340.138551:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145837952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697340.138578:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697340.138582:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697340.138583:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138587:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138591:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138595:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697340.138603:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697340.138617:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145838080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697340.138621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145837952:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697340.138636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697340.138641:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697340.138642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138647:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138650:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138655:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697340.138660:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697340.138672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145838208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697340.138681:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697340.138686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697340.138688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697340.138691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138697:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138701:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138704:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697340.138709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697340.138722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183646680 x1874531145837952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697340.138723:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145838336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697340.138750:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697340.138754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145837952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697340.138755:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697340.138760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138769:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:3.0:1787697340.138769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697340.138776:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697340.138777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.138783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145837952:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697340.138785:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697340.138795:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697340.138809:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145838464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697340.138812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697340.138829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697340.138832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697340.138834:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138839:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138842:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138850:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:3.0:1787697340.138854:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697340.138855:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697340.138869:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145838592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697340.138873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183647480 x1874531145838080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697340.138886:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697340.138893:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697340.138894:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138898:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138902:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138908:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697340.138909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145838080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697340.138915:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697340.138926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697340.138930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145838720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697340.138933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.138941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697340.138947:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697340.138953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838208:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697340.138958:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697340.138962:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697340.138967:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.138971:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.138975:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697340.138979:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697340.138981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697340.138993:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145838848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697340.138998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183644e00 x1874531145838208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697340.139009:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697340.139016:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697340.139018:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697340.139023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145838208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697340.139025:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5239 00000100:00080000:1.0:1787697340.139030:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697340.139037:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787697340.139042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697340.139044:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787697340.139063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697340.139070:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145838976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697340.139071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697340.139085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697340.139111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697340.139123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183644700 x1874531145838336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697340.139135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145838336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697340.139142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697340.139145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.139148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697340.139164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697340.139182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697340.139192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183644000 x1874531145838464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697340.139203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145838464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697340.139210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697340.139213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.139216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145838464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697340.139222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697340.139237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697340.139246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183646680 x1874531145838592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697340.139256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145838592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697340.139270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697340.139273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.139276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697340.139281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697340.139298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697340.139308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183647100 x1874531145838720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697340.139319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145838720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697340.139326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697340.139329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.139332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697340.139338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697340.139380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697340.139392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183647800 x1874531145838848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697340.139402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145838848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697340.139409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697340.139412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.139416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697340.139421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697340.139438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697340.139446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697340/real 1787697340] req@ffff8f5183645180 x1874531145838976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697340.139456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145838976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697340.139462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697340.139465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697340.139467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145838976:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697341.034539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697342.058492:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697343.082566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697345.130626:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697345.258471:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.258481:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697345.258485:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.258494:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258504:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258512:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697345.258547:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145839104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697345.258573:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.258576:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697345.258577:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.258580:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258583:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258586:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258593:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697345.258602:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145839232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697345.258609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.258615:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697345.258618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697345.258621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145839104:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697345.258632:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258637:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258644:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258649:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697345.258662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145839360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697345.258672:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.258674:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697345.258676:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.258679:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258683:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258685:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258689:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697345.258696:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145839488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697345.258703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697345.258711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697345.258712:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697345.258717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.258722:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258726:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258731:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258734:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697345.258745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145839616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697345.258751:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.258754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697345.258755:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.258758:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258761:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697345.258763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183646d80 x1874531145839104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697345.258767:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258779:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697345.258808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145839104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697345.258809:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145839744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697345.258881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697345.258886:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697345.258892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697345.258896:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697345.258899:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697345.258900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145839104:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697345.258905:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00080000:1.0:1787697345.258917:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258923:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258927:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697345.258933:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145839872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697345.258938:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.258940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697345.258943:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.258947:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00100000:3.0:1787697345.258950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839232:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697345.258952:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.258970:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697345.258975:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697345.258989:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145840000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697345.258996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183646a00 x1874531145839232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697345.259013:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697345.259032:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697345.259036:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697345.259045:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5244 00000100:00100000:3.0:1787697345.259051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145839232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697345.259058:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697345.259070:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787697345.259070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697345.259082:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787697345.259087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697345.259099:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145840128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697345.259124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697345.259155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183647b80 x1874531145839360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145839360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697345.259196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697345.259216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183644380 x1874531145839488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145839488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697345.259270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697345.259290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183645c00 x1874531145839616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145839616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145839616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697345.259343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145839744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697345.259384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183645180 x1874531145839744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145839744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145839744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697345.259433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145839872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697345.259453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183647800 x1874531145839872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145839872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145839872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697345.259516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145840000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697345.259537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183647100 x1874531145840000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145840000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145840000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697345.259567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145840128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697345.259582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697345.259592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697345/real 1787697345] req@ffff8f5183646680 x1874531145840128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697345.259603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145840128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697345.259611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697345.259614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697345.259617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145840128:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697346.154518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697347.178507:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697349.226547:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697350.250484:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697350.378460:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378468:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697350.378471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378479:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378487:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378493:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697350.378522:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145840256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697350.378543:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378546:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697350.378547:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378550:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378552:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378554:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378560:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697350.378567:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145840384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697350.378574:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697350.378579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378587:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378590:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697350.378591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145840256:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697350.378598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145840512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697350.378608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697350.378613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378616:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378620:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378624:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378629:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697350.378642:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145840640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697350.378652:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697350.378655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378658:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378661:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378663:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378668:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697350.378671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697350.378680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145840768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697350.378687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697350.378691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378693:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378695:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378698:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697350.378708:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145840896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697350.378711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840512:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697350.378722:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378725:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697350.378726:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378729:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378731:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378733:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697350.378744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697350.378746:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145841024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697350.378754:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697350.378758:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378763:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00100000:3.0:1787697350.378764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840768:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697350.378769:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697350.378773:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378776:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697350.378785:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145841152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697350.378788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145840896:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697350.378799:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697350.378801:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697350.378803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697350.378806:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5250 00000100:00080000:1.0:1787697350.378809:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697350.378810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145841024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697350.378814:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697350.378819:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697350.378828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145841280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697350.378833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145841152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697350.378852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145841280:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697351.274526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697353.322512:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697353.450901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183645c00 x1874531145841280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.450925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145841280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.450937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.450941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.450945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145841280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697353.450952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183645180 x1874531145841152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.450962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145841152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.450970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.450974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.450977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145841152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697353.450983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183647100 x1874531145840896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.450994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145840896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145840896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697353.451016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183647800 x1874531145841024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.451028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145841024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145841024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697353.451048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183644e00 x1874531145840512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.451060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145840512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697353.451080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183647480 x1874531145840640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.451091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145840640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697353.451112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183646680 x1874531145840768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.451123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145840768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697353.451142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183644000 x1874531145840256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.451154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145840256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145840256:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697353.451174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697350/real 1787697353] req@ffff8f5183644700 x1874531145840384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697353.451185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145840384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697353.451194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697353.451198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697353.451208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145840384:192.168.201.125@tcp:38:kworker.0 00010000:00010000:3.0:1787697354.346542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697355.370517:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697355.498671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145841408:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697355.498738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.498801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace93100 x1874531145841408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.498827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145841408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.498847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.498854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.498861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145841408:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697355.498892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697355.498926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.498944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace90e00 x1874531145841536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.498967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145841536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.498989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.498996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697355.499014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697355.499042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace92680 x1874531145841664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145841664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697355.499100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697355.499119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace90000 x1874531145841792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145841792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697355.499155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697355.499170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace93800 x1874531145841920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145841920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145841920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697355.499211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697355.499227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace93b80 x1874531145842048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145842048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697355.499511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697355.499533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace90380 x1874531145842176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145842176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697355.499581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697355.499600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace93100 x1874531145842304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145842304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697355.499638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697355.499653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697355.499662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697355/real 1787697355] req@ffff8f51ace90700 x1874531145842432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697355.499670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145842432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697355.499681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697355.499685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697355.499688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145842432:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697357.418502:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697358.442538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697359.466516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697360.618636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145842560:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697360.618732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.618779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace91c00 x1874531145842560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.618808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145842560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.618831:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.618839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.618848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145842560:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697360.618880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145842688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697360.618911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.618930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace91880 x1874531145842688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.618953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145842688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.618984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.618990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.618999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145842688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697360.619009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145842816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697360.619038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace91500 x1874531145842816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145842816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145842816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697360.619112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145842944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697360.619143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace92300 x1874531145842944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145842944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145842944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697360.619187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145843072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697360.619207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace93480 x1874531145843072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145843072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145843072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697360.619272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697360.619291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace92d80 x1874531145843200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145843200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697360.619332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697360.619351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace91180 x1874531145843328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145843328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697360.619425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697360.619443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace91c00 x1874531145843456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145843456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697360.619482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697360.619495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697360.619502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697360/real 1787697360] req@ffff8f51ace90700 x1874531145843584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697360.619508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145843584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697360.619513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697360.619515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697360.619517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145843584:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697361.514532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697362.538554:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697363.562578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697365.610556:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697365.738539:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697365.738550:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697365.738555:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.738566:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.738586:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.738593:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697365.738676:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145843712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697365.738717:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697365.738722:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697365.738724:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.738729:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.738734:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.738738:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697365.738750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697365.738765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145843840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697365.738776:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697365.738781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145843712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697365.738790:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697365.738793:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.738799:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.738803:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.738807:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697365.738812:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697365.738824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145843968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697365.738834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697365.738837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697365.738840:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.738844:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.738848:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.738855:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:3.0:1787697365.738855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697365.738864:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697365.738880:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145844096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697365.738890:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697365.738893:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697365.738895:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.738900:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.738904:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697365.738909:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183644380 x1874531145843712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697365.738913:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697365.738921:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697365.738936:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145844224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697365.738938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145843712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697365.738966:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697365.738966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.738973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697365.738974:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787697365.738978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145843712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697365.738979:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.738989:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.738993:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.738997:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697365.739003:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697365.739007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145843840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697365.739019:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145844352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697365.739029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697365.739042:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697365.739043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183647b80 x1874531145843840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697365.739052:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697365.739057:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697365.739061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145843840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697365.739065:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00080000:1.0:1787697365.739072:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.739080:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787697365.739085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697365.739088:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697365.739092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145843840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697365.739105:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145844480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697365.739108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145843968:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697365.739129:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697365.739133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697365.739136:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697365.739140:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.739146:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00100000:3.0:1787697365.739151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183646a00 x1874531145843968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697365.739152:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.739161:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697365.739169:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697365.739174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145843968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697365.739186:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145844608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697365.739199:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697365.739202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697365.739203:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697365.739206:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697365.739212:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5265 00000100:00100000:3.0:1787697365.739216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145843968:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697365.739218:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697365.739226:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145844096:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697365.739233:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697365.739249:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145844736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697365.739260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183646d80 x1874531145844096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697365.739285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145844096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.739297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145844096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697365.739307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145844224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697365.739327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183644700 x1874531145844224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697365.739350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145844224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.739392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145844224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697365.739412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697365.739431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183644000 x1874531145844352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697365.739451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145844352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.739463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697365.739471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697365.739489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183646680 x1874531145844480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697365.739510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145844480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.739519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697365.739525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697365.739539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183647480 x1874531145844608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697365.739564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145844608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.739575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697365.739584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697365.739603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697365.739625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697365/real 1787697365] req@ffff8f5183644e00 x1874531145844736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697365.739636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145844736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697365.739645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697365.739648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697365.739651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145844736:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697366.634532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697367.658509:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697369.706512:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697370.730508:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697370.860596:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.860606:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697370.860611:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.860625:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.860636:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.860643:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697370.860680:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145844864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697370.860707:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.860711:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697370.860713:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.860717:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.860721:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.860726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697370.860735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697370.860742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145844864:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697370.860749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145844992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697370.860761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.860767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697370.860769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.860772:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.860778:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.860782:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697370.860786:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697370.860798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697370.860801:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145845120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697370.860813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.860815:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697370.860817:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.860821:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.860825:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.860828:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697370.860834:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697370.860834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183647800 x1874531145844864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697370.860851:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145845248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697370.860854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145844864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697370.860868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.860876:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697370.860879:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.860885:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:3.0:1787697370.860886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697370.860890:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697370.860895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697370.860897:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697370.860903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145844864:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697370.860905:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697370.860920:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145845376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697370.860930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145844992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697370.860932:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.860939:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697370.860940:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.860945:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.860949:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.860954:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697370.860959:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787697370.860959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697370.860975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183647100 x1874531145844992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697370.860976:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145845504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697370.861001:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.861005:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787697370.861008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145844992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697370.861009:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.861018:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.861024:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697370.861027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697370.861031:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787697370.861033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145844992:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697370.861039:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697370.861043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145845120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697370.861056:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145845632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697370.861066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697370.861068:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697370.861073:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697370.861075:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697370.861080:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00100000:3.0:1787697370.861083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183645180 x1874531145845120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697370.861085:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.861093:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697370.861101:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697370.861106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145845120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697370.861115:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145845760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697370.861118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697370.861132:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697370.861133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145845120:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697370.861139:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697370.861143:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697370.861145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145845248:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697370.861149:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5270 00000100:00080000:1.0:1787697370.861155:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697370.861159:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697370.861164:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697370.861170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697370.861178:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145845888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697370.861193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183645c00 x1874531145845248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697370.861209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145845248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.861216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145845248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697370.861237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145845376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697370.861265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697370.861277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183644e00 x1874531145845376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697370.861287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145845376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.861295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145845376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697370.861307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697370.861324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697370.861334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183647480 x1874531145845504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697370.861345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145845504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.861373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697370.861385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697370.861404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697370.861415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183646680 x1874531145845632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697370.861426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145845632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.861433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697370.861444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697370.861459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697370.861467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183644000 x1874531145845760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697370.861478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145845760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.861484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697370.861495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697370.861512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697370.861536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff8f5183644700 x1874531145845888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697370.861547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145845888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.861555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697370.861558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697370.861561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145845888:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697371.754550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697373.802521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697374.826544:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697375.850522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697375.978521:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978536:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697375.978541:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978554:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978568:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978577:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697375.978622:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145846016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978662:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697375.978663:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978668:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978672:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978676:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978686:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697375.978706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145846144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978717:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697375.978720:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978726:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978728:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697375.978740:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145846272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978746:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978749:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697375.978750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978752:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978754:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978756:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697375.978766:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145846400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978776:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978779:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697375.978780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978783:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978787:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978789:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978792:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697375.978798:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145846528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697375.978807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978809:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978813:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978816:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697375.978823:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145846656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978828:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978833:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697375.978834:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978847:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978850:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978853:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697375.978862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145846784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978875:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978881:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697375.978884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978888:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978890:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978894:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978900:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697375.978914:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145846912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697375.978923:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.978927:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697375.978929:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.978933:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5275 00000100:00080000:3.0:1787697375.978937:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697375.978950:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697375.978955:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697375.978969:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145847040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697377.898559:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697378.922572:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697379.946599:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697381.098481:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098491:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697381.098495:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098506:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098517:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098523:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697381.098562:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145847168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098592:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697381.098593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098599:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098609:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697381.098617:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145847296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098623:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098626:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697381.098627:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098630:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098631:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098634:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098636:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697381.098644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145847424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697381.098658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098660:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098661:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098665:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098669:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697381.098677:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145847552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098683:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098685:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697381.098686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098688:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098690:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098692:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098695:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697381.098701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145847680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697381.098709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098712:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098715:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098718:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098721:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697381.098727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145847808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098732:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098734:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697381.098736:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098738:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098741:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098743:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697381.098753:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145847936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098762:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697381.098764:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098767:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098776:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098779:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098784:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697381.098790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145848064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697381.098800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697381.098802:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697381.098804:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697381.098806:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5280 00000100:00080000:3.0:1787697381.098808:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697381.098810:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697381.098814:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697381.098827:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145848192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697381.994579:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697383.018554:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697384.042512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697386.090629:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697386.218458:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697386.218471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218480:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218491:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218506:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697386.218547:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145848320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218570:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218573:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697386.218574:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218577:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218580:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218583:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218589:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697386.218598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145848448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218605:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218607:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697386.218608:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218610:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218612:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218616:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218619:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697386.218627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145848576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218632:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697386.218635:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218636:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218639:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218641:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218644:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697386.218652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145848704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218659:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697386.218661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218664:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218681:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218685:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697386.218705:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145848832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218715:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218718:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697386.218719:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218726:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218730:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697386.218749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145848960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218767:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697386.218777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218784:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218791:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218799:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218805:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697386.218819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145849088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697386.218833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218836:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218840:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218846:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697386.218862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145849216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697386.218891:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697386.218895:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697386.218899:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697386.218908:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5285 00000100:00080000:1.0:1787697386.218914:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697386.218920:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697386.218926:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697386.218938:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145849344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697387.114532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697388.138502:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697390.186605:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697391.210583:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697391.338784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145849472:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697391.338876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.338929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace93100 x1874531145849472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.338959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145849472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.338974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.338982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.338990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145849472:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697391.339024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697391.339052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace90380 x1874531145849600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145849600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697391.339117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697391.339145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace93b80 x1874531145849728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145849728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697391.339232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697391.339261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace93800 x1874531145849856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145849856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697391.339348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697391.339384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace90000 x1874531145849984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145849984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145849984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697391.339411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697391.339422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace92680 x1874531145850112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145850112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697391.339483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697391.339503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace90e00 x1874531145850240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145850240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697391.339546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697391.339564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace93100 x1874531145850368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145850368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697391.339611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697391.339631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697391.339641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697391/real 1787697391] req@ffff8f51ace90700 x1874531145850496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697391.339652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145850496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697391.339660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697391.339663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697391.339667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145850496:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697392.234659:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697394.282673:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697395.306609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697396.331346:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697396.458481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458492:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697396.458497:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458508:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458519:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458526:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697396.458568:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145850624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697396.458601:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697396.458620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458625:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458630:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458634:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.458645:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697396.458645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145850624:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697396.458663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145850752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697396.458673:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458677:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697396.458678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458682:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458685:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458687:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.458691:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697396.458700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145850880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697396.458709:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697396.458714:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458718:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458722:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458725:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.458731:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697396.458733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697396.458747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145851008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697396.458757:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458761:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697396.458762:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458767:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458771:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458775:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.458780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697396.458787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183645c00 x1874531145850624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697396.458797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145851136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697396.458809:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697396.458813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145850624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697396.458814:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697396.458817:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458828:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458834:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458842:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787697396.458846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697396.458849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697396.458854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.458861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145850624:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697396.458866:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145851264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697396.458876:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697396.458881:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.458899:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458903:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697396.458905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145850752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697396.458910:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.458917:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697396.458931:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145851392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697396.458939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697396.458947:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697396.458952:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697396.458954:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697396.458959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183645180 x1874531145850752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697396.458960:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00080000:1.0:1787697396.458966:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.458974:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.458982:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697396.458984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145850752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.458995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697396.458997:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145851520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697396.459003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697396.459010:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697396.459012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145850752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697396.459014:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697396.459018:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697396.459023:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5296 00000100:00100000:3.0:1787697396.459026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145850880:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697396.459027:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697396.459033:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697396.459038:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697396.459047:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145851648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697396.459060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183647100 x1874531145850880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145850880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145850880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697396.461233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145851008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697396.461272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183647800 x1874531145851008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145851008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145851008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697396.461318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145851136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697396.461338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183644e00 x1874531145851136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145851136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145851136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697396.461398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697396.461419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183647480 x1874531145851264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145851264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697396.461490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697396.461509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183646680 x1874531145851392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145851392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697396.461553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697396.461569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183644000 x1874531145851520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145851520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697396.461615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697396.461632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697396.461642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697396/real 1787697396] req@ffff8f5183644700 x1874531145851648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697396.461652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145851648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697396.461660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697396.461663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697396.461667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145851648:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697398.378494:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697399.402640:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697400.426513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697401.578428:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578438:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697401.578442:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578451:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578461:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578468:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697401.578502:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145851776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578540:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578544:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697401.578546:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578550:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578554:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578562:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578571:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697401.578583:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145851904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697401.578597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578601:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578604:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578608:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578613:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697401.578623:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145852032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578631:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578634:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697401.578636:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578639:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578642:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578645:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578650:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697401.578659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145852160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578667:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697401.578671:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578675:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578678:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578681:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578685:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697401.578697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145852288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578709:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697401.578710:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578713:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578716:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697401.578735:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145852416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578748:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578751:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697401.578753:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578756:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578760:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578763:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578767:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697401.578778:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145852544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578785:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578788:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697401.578790:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578793:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578796:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578799:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578804:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697401.578813:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145852672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697401.578820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697401.578823:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697401.578824:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697401.578827:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5301 00000100:00080000:1.0:1787697401.578831:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697401.578834:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697401.578838:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697401.578846:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145852800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697401.578926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145851776:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787697401.578983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183644380 x1874531145851776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145851776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145851776:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697401.579080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145851904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697401.579103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183647b80 x1874531145851904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145851904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145851904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697401.579144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145852032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697401.579159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183646a00 x1874531145852032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145852032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145852032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697401.579184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145852160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697401.579193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183646d80 x1874531145852160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145852160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145852160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697401.579229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145852288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697401.579246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183644700 x1874531145852288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145852288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145852288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697401.579286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697401.579305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183644000 x1874531145852416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145852416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697401.579346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697401.579391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183646680 x1874531145852544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145852544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697401.579432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697401.579465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183647480 x1874531145852672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145852672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697401.579514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697401.579532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697401.579539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697401/real 1787697401] req@ffff8f5183644e00 x1874531145852800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697401.579546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145852800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697401.579552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697401.579553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697401.579555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145852800:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697402.474541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697403.498750:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697404.522529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697406.570529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697406.698519:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698530:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697406.698534:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698543:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698554:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698562:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697406.698596:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145852928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698623:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698627:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697406.698628:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698633:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698640:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.698648:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697406.698661:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145853056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697406.698683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698687:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698691:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698694:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.698699:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697406.698710:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145853184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698719:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698722:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697406.698724:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698731:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698737:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698741:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.698747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697406.698759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145853312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698769:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698778:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697406.698782:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698790:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698801:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.698809:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697406.698824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145853440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698840:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698844:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697406.698846:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698853:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698859:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698864:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.698869:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697406.698890:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145853568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698919:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698925:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697406.698927:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.698935:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.698943:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.698950:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.698959:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697406.698978:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145853696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.698989:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.698996:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697406.699000:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.699018:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.699022:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.699027:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.699036:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697406.699053:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145853824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697406.699073:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697406.699080:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697406.699084:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697406.699089:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5306 00000100:00080000:3.0:1787697406.699094:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697406.699102:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697406.699109:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697406.699125:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145853952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697407.594533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697408.618526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697410.666517:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697411.690514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697411.818471:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818481:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697411.818484:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818494:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818516:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818524:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697411.818553:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145854080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818574:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818576:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697411.818578:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818581:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818586:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818592:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697411.818601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145854208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697411.818612:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818614:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818615:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818617:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697411.818633:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145854336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818642:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697411.818642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818644:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818646:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818648:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818651:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697411.818658:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145854464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818664:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697411.818667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818672:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818674:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697411.818681:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145854592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697411.818689:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818692:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818694:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818699:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818705:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697411.818711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145854720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818717:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818720:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697411.818721:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818725:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818727:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818730:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697411.818739:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145854848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818763:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818768:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697411.818770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818775:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818780:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818786:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818794:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697411.818810:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145854976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697411.818854:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697411.818857:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697411.818859:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697411.818864:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5311 00000100:00080000:3.0:1787697411.818868:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697411.818872:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697411.818875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697411.818885:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145855104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697412.714487:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697414.762517:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697415.786522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697416.810528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697416.938466:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938475:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697416.938479:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938489:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938499:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938506:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697416.938553:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145855232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938585:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938588:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697416.938590:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938594:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938610:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697416.938622:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145855360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938633:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938637:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697416.938640:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938645:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938648:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938653:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938661:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697416.938673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145855488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697416.938684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938686:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938688:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938690:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938694:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697416.938702:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145855616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938711:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938714:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697416.938716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938719:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938737:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938742:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697416.938758:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145855744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938767:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938770:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697416.938772:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938784:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938790:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697416.938807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145855872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938818:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938820:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697416.938822:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938826:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938832:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938840:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938846:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697416.938858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145856000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938868:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697416.938873:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938879:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938886:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938890:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938895:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697416.938908:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145856128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697416.938930:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697416.938934:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697416.938936:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697416.938941:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5316 00000100:00080000:3.0:1787697416.938949:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697416.938955:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697416.938960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697416.938987:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145856256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697418.858604:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697419.882466:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697420.011795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f518738b480 x1874531145856256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.011828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145856256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.011839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.011843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.011847:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145856256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697420.011855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f518738a300 x1874531145855872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.011866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145855872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.011879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.011882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.011884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145855872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697420.011891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f518738bb80 x1874531145856000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.011901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145856000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.011908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.011911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.011913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145856000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697420.011919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f5187389c00 x1874531145856128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.011933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145856128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.011939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.011942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.011945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145856128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697420.011950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f518738ad80 x1874531145855360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.011960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145855360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.011967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.011971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.011973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145855360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697420.011979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f5187389500 x1874531145855488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.011988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145855488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.011993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.011996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.011999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145855488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697420.012004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f518738aa00 x1874531145855616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.012012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145855616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.012018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.012021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.012023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145855616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697420.012029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f5187389180 x1874531145855744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.012037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145855744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.012044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.012047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.012049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145855744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697420.012054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697420] req@ffff8f518738b800 x1874531145855232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697420.012064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145855232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697420.012070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697420.012072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697420.012075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145855232:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787697420.906522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697422.058496:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058510:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697422.058515:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058527:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058541:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058549:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697422.058601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145856384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697422.058655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058661:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058666:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058670:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.058682:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697422.058697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145856512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058711:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697422.058718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058726:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058730:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.058735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697422.058748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145856640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058761:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697422.058763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058767:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058771:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058775:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.058780:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697422.058792:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145856768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058808:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058811:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697422.058814:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058819:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058823:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058827:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.058832:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697422.058844:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145856896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058854:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058862:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697422.058866:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058881:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058885:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058892:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.058902:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697422.058919:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145857024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058938:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.058941:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697422.058943:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.058948:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.058953:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.058961:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.058970:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697422.058984:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145857152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.058997:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.059002:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697422.059004:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.059011:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.059017:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.059025:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.059033:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697422.059053:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145857280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697422.059072:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697422.059078:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697422.059080:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697422.059084:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5321 00000100:00080000:3.0:1787697422.059090:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697422.059095:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697422.059102:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697422.059124:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145857408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1787697422.954585:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697423.978643:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697425.002627:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697427.050480:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697427.178450:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178459:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697427.178462:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178469:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178481:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178486:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697427.178514:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145857536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178536:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178539:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697427.178540:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178543:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178555:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178560:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178567:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697427.178581:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145857664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178592:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178595:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697427.178600:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178611:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178613:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178617:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697427.178626:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145857792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178633:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178635:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697427.178636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178638:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178640:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178642:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178645:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697427.178655:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145857920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178663:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178665:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697427.178667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178670:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178672:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178674:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697427.178684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145858048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178691:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697427.178699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178706:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178709:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178711:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697427.178724:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145858176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178736:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697427.178743:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178747:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178749:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178751:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697427.178759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145858304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178775:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178778:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697427.178779:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178783:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178786:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178791:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178795:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697427.178805:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145858432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697427.178843:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697427.178850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697427.178854:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697427.178862:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5326 00000100:00080000:3.0:1787697427.178866:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697427.178870:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697427.178877:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697427.178889:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145858560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697427.179076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697427.179078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145858432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697427.179081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145858560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697427.179089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697427.179093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697427/real 1787697427] req@ffff8f518738b480 x1874531145858560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697427.179099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145858560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697427.179104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697427.179105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697427.179106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145858560:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697428.074489:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697429.098613:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697430.122491:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697431.146553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697432.170505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697432.298479:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298486:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697432.298489:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298497:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298505:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298510:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697432.298539:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145858688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697432.298560:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298562:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697432.298564:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298566:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298568:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298570:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298577:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697432.298584:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145858816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697432.298590:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298592:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697432.298593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298598:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298600:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298604:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697432.298612:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145858944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697432.298618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697432.298621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298624:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298626:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298628:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697432.298635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145858688:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697432.298640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145859072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697432.298647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697432.298650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298652:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298654:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298655:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697432.298664:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145859200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697432.298669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298671:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697432.298672:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298675:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298677:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298679:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298682:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697432.298686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697432.298692:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145859328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697432.298703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298705:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697432.298706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298709:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298711:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298714:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298717:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697432.298720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f518738a300 x1874531145858688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697432.298731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145859456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697432.298734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145858688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.298757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697432.298759:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697432.298762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697432.298767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697432.298768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145858688:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697432.298770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697432.298777:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00080000:3.0:1787697432.298781:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298784:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697432.298786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145858816:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697432.298787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697432.298801:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145859584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697432.298810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697432.298811:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697432.298816:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697432.298818:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697432.298821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f518738bb80 x1874531145858816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697432.298826:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5331 00000100:00100000:0.0:1787697432.298833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145858816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697432.298840:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697432.298848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697432.298855:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697432.298859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.298865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697432.298867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145859712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697432.298874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145858816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697432.298884:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145858944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697432.298907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.298918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f5187389c00 x1874531145858944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.298929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145858944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.298937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.298940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.298943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145858944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697432.298966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145859072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697432.298991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.299001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f518738ad80 x1874531145859072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.299018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145859072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.299026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.299029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.299032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145859072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697432.299038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145859200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697432.299060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.299068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f5187389500 x1874531145859200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.299074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145859200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.299079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.299081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.299083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145859200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697432.299086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697432.299095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.299101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f518738aa00 x1874531145859328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.299107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145859328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.299118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.299121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.299124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697432.299129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697432.299146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.299156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f5187389180 x1874531145859456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.299166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145859456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.299173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.299176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.299179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697432.299185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697432.299204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.299215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f518738b800 x1874531145859584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.299226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145859584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.299234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.299236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.299240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697432.299246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697432.299262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697432.299269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697432/real 1787697432] req@ffff8f518738b480 x1874531145859712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697432.299276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145859712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697432.299281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697432.299282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697432.299284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145859712:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697433.194548:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697434.218510:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697435.242541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697436.266533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697437.290550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697437.418479:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418488:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697437.418491:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418500:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418509:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418515:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697437.418552:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145859840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697437.418576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418580:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697437.418581:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418585:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418589:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418594:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697437.418602:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697437.418620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145859968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697437.418627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145859840:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697437.418630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418637:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697437.418639:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418641:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418644:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418646:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697437.418649:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697437.418658:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145860096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697437.418664:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697437.418667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418683:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418685:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697437.418687:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697437.418695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697437.418696:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145860224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697437.418703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418706:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697437.418707:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418710:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418712:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418714:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697437.418718:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697437.418724:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145860352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697437.418730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418732:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697437.418733:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418735:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418738:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418742:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697437.418745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183647800 x1874531145859840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697437.418745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697437.418755:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145860480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697437.418765:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418768:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697437.418769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697437.418774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145859840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697437.418774:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418778:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418782:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697437.418793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697437.418793:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697437.418801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697437.418804:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145860608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697437.418809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145859840:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697437.418814:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418817:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697437.418819:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418823:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418825:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418826:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697437.418828:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697437.418833:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145860736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697437.418840:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697437.418843:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697437.418848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145859968:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697437.418853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697437.418862:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5337 00000100:00080000:1.0:1787697437.418867:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697437.418870:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697437.418875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697437.418883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697437.418888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145860864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697437.418899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183647100 x1874531145859968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.418925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145859968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.418934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.418937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.418940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145859968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697437.418946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145860096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697437.418966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.418977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183645180 x1874531145860096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145860096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145860096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697437.419021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145860224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697437.419041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.419066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183645c00 x1874531145860224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145860224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145860224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697437.419104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145860352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697437.419122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.419132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183644e00 x1874531145860352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145860352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145860352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697437.419162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697437.419181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.419191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183647480 x1874531145860480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145860480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697437.419235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697437.419255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.419266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183646680 x1874531145860608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145860608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697437.419295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697437.419312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.419323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183644000 x1874531145860736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145860736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697437.419418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697437.419441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697437.419454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697437/real 1787697437] req@ffff8f5183644700 x1874531145860864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697437.419466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145860864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697437.419473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697437.419476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697437.419479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145860864:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697438.314527:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697439.338550:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697440.362527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697441.386538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697442.410623:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697442.538483:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538496:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697442.538501:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538514:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538526:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538534:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697442.538579:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145860992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697442.538610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697442.538615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538620:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538624:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538628:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538639:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697442.538652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145861120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697442.538663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538668:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697442.538670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145860992:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697442.538671:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538679:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538683:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538687:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538692:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697442.538706:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145861248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697442.538716:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538720:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697442.538722:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538725:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538730:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538734:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697442.538751:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145861376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697442.538757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697442.538766:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538770:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697442.538774:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538779:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538783:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538787:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697442.538804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145861504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697442.538814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f5187389f80 x1874531145860992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697442.538815:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538824:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697442.538828:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538837:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538840:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538845:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538850:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697442.538854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145860992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697442.538865:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145861632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697442.538869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.538875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697442.538875:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538881:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697442.538884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145860992:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697442.538885:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538892:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538895:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538898:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538901:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697442.538909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145861760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697442.538914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861120:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697442.538917:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538922:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697442.538923:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538927:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00080000:3.0:1787697442.538928:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538931:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697442.538942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697442.538944:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145861888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697442.538951:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697442.538954:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697442.538955:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697442.538960:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5342 00000100:00100000:0.0:1787697442.538961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f5187388000 x1874531145861120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697442.538964:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697442.538971:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697442.538980:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697442.538985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145861120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697442.538995:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145862016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697442.538998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697442.539018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697442.539035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f518738a680 x1874531145861248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145861248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697442.539062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697442.539072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f5187388380 x1874531145861376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145861376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697442.539117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697442.539137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f518738b100 x1874531145861504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145861504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145861504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697442.539166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145861632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697442.539177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f5187388a80 x1874531145861632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145861632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145861632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697442.539213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145861760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697442.539223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f5187388e00 x1874531145861760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145861760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145861760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697442.539258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145861888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697442.539276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f5187389f80 x1874531145861888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145861888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145861888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697442.539331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145862016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697442.539346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697442.539384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697442/real 1787697442] req@ffff8f518738b480 x1874531145862016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697442.539396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145862016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697442.539403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697442.539404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697442.539406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145862016:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697443.434569:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697444.458557:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697445.482571:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697446.506612:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697447.530598:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697447.660603:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.660614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697447.660617:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.660627:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.660638:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697447.660645:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697447.660692:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145862144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697447.660722:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.660726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697447.660728:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.660733:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.660738:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697447.660742:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.660754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697447.660766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145862144:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697447.660770:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145862272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697447.660784:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.660787:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697447.660790:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.660794:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.660798:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697447.660801:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.660807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697447.660819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145862400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697447.660829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.660832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697447.660834:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.660838:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.660841:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697447.660845:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.660850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697447.660862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145862528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697447.660872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697447.660883:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.660886:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697447.660888:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.660893:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.660896:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697447.660899:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.660904:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697447.660916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145862656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697447.660926:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.660929:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697447.660931:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.660936:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.660941:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697447.660944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace91c00 x1874531145862144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697447.660948:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.660956:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697447.660971:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145862784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697447.660974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145862144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697447.660997:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697447.660999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697447.661005:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697447.661009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697447.661010:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.661019:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00100000:3.0:1787697447.661019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145862144:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697447.661027:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697447.661032:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.661038:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697447.661050:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145862912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697447.661054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862272:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697447.661069:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697447.661072:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697447.661074:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.661078:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.661082:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697447.661087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697447.661088:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.661095:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697447.661109:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145863040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697447.661117:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697447.661118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace91180 x1874531145862272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697447.661124:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697447.661128:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697447.661136:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5347 00000100:00080000:0.0:1787697447.661141:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697447.661146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145862272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697447.661147:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697447.661154:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787697447.661164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697447.661172:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145863168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697447.661177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697447.661191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697447.661244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace92d80 x1874531145862400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145862400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697447.661294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697447.661316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace93480 x1874531145862528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145862528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697447.661394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697447.661413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace92300 x1874531145862656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145862656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145862656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697447.661471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145862784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697447.661490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace91500 x1874531145862784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145862784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145862784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697447.661544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145862912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697447.661562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace91880 x1874531145862912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145862912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145862912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697447.661605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145863040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697447.661622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace91c00 x1874531145863040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145863040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145863040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697447.661670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145863168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697447.661687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697447.661697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697447/real 1787697447] req@ffff8f51ace90700 x1874531145863168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697447.661708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145863168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697447.661716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697447.661718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697447.661721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145863168:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697448.554608:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697449.578573:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697450.602555:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697451.626503:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697452.650575:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697452.778506:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778517:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697452.778522:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.778535:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.778546:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778554:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697452.778597:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145863296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697452.778630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697452.778636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.778641:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.778645:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778650:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.778662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697452.778672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145863296:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697452.778678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145863424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697452.778690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697452.778695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.778700:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.778704:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778708:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.778715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697452.778727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145863552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697452.778737:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697452.778742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697452.778748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697452.778751:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.778757:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778762:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.778778:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697452.778790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145863680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697452.778800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f518738b800 x1874531145863296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697452.778802:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778810:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697452.778813:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.778821:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.778825:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778829:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697452.778834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145863296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697452.778834:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697452.778848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697452.778849:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145863808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697452.778855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.778862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145863296:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697452.778867:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697452.778874:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.778878:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.778881:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778884:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.778889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697452.778890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697452.778901:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145863936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697452.778911:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697452.778915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697452.778917:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697452.778920:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.778925:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00100000:0.0:1787697452.778929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f5187389180 x1874531145863424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697452.778930:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.778937:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.778944:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697452.778946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145863424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.778953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.778957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.778959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697452.778965:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145864064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697452.778966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697452.778985:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.778989:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697452.778991:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697452.778992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697452.778996:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.779001:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697452.779003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f518738aa00 x1874531145863552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697452.779006:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.779013:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697452.779018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145863552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697452.779026:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145864192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697452.779027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697452.779042:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697452.779045:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697452.779046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863680:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697452.779047:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697452.779053:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5352 00000100:00080000:3.0:1787697452.779055:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697452.779058:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697452.779061:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697452.779068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697452.779071:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145864320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697452.779085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f5187389500 x1874531145863680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697452.779099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145863680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.779127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697452.779137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697452.779152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697452.779160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f518738ad80 x1874531145863808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697452.779172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145863808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.779177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145863808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697452.779191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145863936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697452.779208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697452.779218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f5187389c00 x1874531145863936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697452.779230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145863936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.779238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145863936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697452.779250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145864064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697452.779268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697452.779278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f518738bb80 x1874531145864064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697452.779289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145864064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.779302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145864064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697452.779315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145864192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697452.779331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697452.779338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f518738a300 x1874531145864192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697452.779345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145864192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.779350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145864192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697452.779392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145864320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697452.779412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697452.779422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697452/real 1787697452] req@ffff8f518738b800 x1874531145864320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697452.779430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145864320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697452.779435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697452.779437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697452.779439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145864320:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697453.675280:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697454.698629:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697455.723781:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697456.746652:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697457.770548:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697457.898459:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898469:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697457.898473:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697457.898482:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697457.898500:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697457.898534:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145864448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697457.898560:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898564:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697457.898566:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697457.898570:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898574:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697457.898578:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898587:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697457.898601:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145864576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697457.898611:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898614:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697457.898616:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697457.898619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145864448:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697457.898621:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898629:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697457.898634:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898638:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697457.898653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145864704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697457.898663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898666:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697457.898667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697457.898672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697457.898679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697457.898683:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898689:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697457.898701:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145864832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697457.898710:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898713:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697457.898715:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697457.898718:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898721:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697457.898724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f518738b480 x1874531145864448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697457.898727:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898734:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697457.898746:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145864960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697457.898750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145864448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697457.898763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697457.898767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697457.898767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.898774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697457.898775:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898779:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697457.898780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145864448:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697457.898785:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697457.898796:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145865088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697457.898802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864576:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697457.898803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898808:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697457.898809:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697457.898813:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898816:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697457.898818:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898822:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697457.898826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.898838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f5187389f80 x1874531145864576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697457.898838:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145865216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697457.898850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145864576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.898868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.898874:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697457.898874:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697457.898879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864576:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697457.898882:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697457.898884:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697457.898886:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864704:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697457.898890:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898897:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697457.898900:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697457.898904:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697457.898908:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.898921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f5187388e00 x1874531145864704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697457.898922:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145865344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697457.898946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145864704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697457.898947:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697457.898953:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697457.898956:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.898962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697457.898962:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5357 00000100:00080000:3.0:1787697457.898968:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697457.898969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.898974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864704:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697457.898977:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697457.898982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864832:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787697457.898985:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697457.899000:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145865472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697457.899002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.899026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f5187388a80 x1874531145864832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697457.899040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145864832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.899048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.899052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.899055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697457.899061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697457.899075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.899082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f518738b100 x1874531145864960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697457.899094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145864960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.899098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.899100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.899102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145864960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697457.899112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697457.899122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.899127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f5187388380 x1874531145865088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697457.899134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145865088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.899139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.899140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.899142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697457.899145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697457.899153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.899158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f518738a680 x1874531145865216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697457.899164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145865216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.899172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.899174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.899175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697457.899178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697457.899186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.899191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f5187388000 x1874531145865344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697457.899197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145865344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.899200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.899202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.899203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697457.899206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697457.899213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697457.899218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697457/real 1787697457] req@ffff8f518738b480 x1874531145865472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697457.899224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145865472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.899229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697457.899230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697457.899232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145865472:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697458.794581:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697459.818533:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697460.842653:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697461.866611:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697462.890519:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697463.018448:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018458:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697463.018463:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018473:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018485:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018493:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697463.018524:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145865600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697463.018554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018557:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018559:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018562:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697463.018569:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697463.018577:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145865728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018592:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697463.018594:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697463.018598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145865600:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697463.018600:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018606:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018611:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697463.018616:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697463.018630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145865856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018639:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697463.018643:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018646:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018658:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018662:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697463.018666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697463.018670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697463.018681:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145865984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697463.018694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018697:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018701:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018705:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697463.018709:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697463.018715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183646d80 x1874531145865600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697463.018723:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145866112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018735:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697463.018740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145865600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697463.018740:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697463.018744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018751:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018755:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018761:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697463.018763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697463.018764:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697463.018772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.018779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145865600:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697463.018779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145866240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018797:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697463.018802:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018807:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00100000:0.0:1787697463.018810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145865728:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697463.018811:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697463.018819:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697463.018824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697463.018834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697463.018838:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145866368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697463.018847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183646a00 x1874531145865728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697463.018850:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697463.018863:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697463.018866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145865728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697463.018870:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018877:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697463.018879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.018885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697463.018885:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697463.018891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145865728:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697463.018894:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697463.018897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145865856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697463.018909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145866496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697463.018918:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697463.018920:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697463.018922:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697463.018926:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5362 00000100:00080000:1.0:1787697463.018943:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697463.018946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697463.018951:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697463.018957:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697463.018964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183647b80 x1874531145865856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697463.018970:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145866624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697463.018986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145865856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.018995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.018998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145865856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697463.019008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145865984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697463.019029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697463.019039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183644380 x1874531145865984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697463.019051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145865984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.019057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.019060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145865984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697463.019068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145866112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697463.019086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697463.019096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183644700 x1874531145866112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697463.019112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145866112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.019119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.019122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145866112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697463.019131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697463.019150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697463.019161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183644000 x1874531145866240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697463.019173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145866240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.019181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.019184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697463.019206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697463.019225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697463.019237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183646680 x1874531145866368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697463.019255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145866368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.019265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.019267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697463.019274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697463.019288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697463.019295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183647480 x1874531145866496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697463.019301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145866496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.019305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.019307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697463.019311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697463.019325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697463.019333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697463/real 1787697463] req@ffff8f5183644e00 x1874531145866624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697463.019343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145866624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697463.019350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697463.019384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697463.019390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145866624:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697463.914550:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697464.938537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697465.962586:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697466.986593:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697468.010549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697468.138509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138516:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697468.138519:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138527:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138535:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.138540:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697468.138567:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145866752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697468.138590:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138592:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697468.138593:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138596:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138598:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.138600:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.138606:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697468.138614:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145866880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697468.138620:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138622:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697468.138624:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138627:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138629:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.138630:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.138633:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697468.138643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145866752:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697468.138643:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145867008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697468.138654:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697468.138657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138660:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138664:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.138667:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.138672:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697468.138683:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145867136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697468.138692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138696:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787697468.138697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697468.138699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138706:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138710:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.138714:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.138718:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697468.138728:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145867264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697468.138733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f518738b800 x1874531145866752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697468.138736:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697468.138746:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138751:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138753:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697468.138755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145866752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.138763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697468.138765:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697468.138768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697468.138771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697468.138773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145866752:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697468.138782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145866880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697468.138782:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145867392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697468.138794:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.138797:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697468.138798:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.138802:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.138806:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697468.138808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697468.138816:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.138822:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697468.138824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f518738a300 x1874531145866880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697468.139083:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145867520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697468.139085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145866880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697468.139101:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.139104:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697468.139105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697468.139108:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697468.139112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697468.139114:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00100000:0.0:1787697468.139120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145866880:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697468.139120:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.139125:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.139131:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697468.139149:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145867648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697468.139151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145867008:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697468.139171:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697468.139174:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697468.139175:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697468.139180:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5367 00000100:00080000:3.0:1787697468.139183:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697468.139187:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697468.139192:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697468.139198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697468.139204:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145867776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697468.139215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f518738bb80 x1874531145867008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145867008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145867008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697468.139245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145867136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697468.139260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697468.139267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f5187389c00 x1874531145867136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145867136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145867136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697468.139289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145867264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697468.139304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697468.139313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f518738ad80 x1874531145867264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145867264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145867264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697468.139411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697468.139433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697468.139443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f5187389500 x1874531145867392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145867392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697468.139472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697468.139488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697468.139497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f518738aa00 x1874531145867520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145867520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697468.139541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697468.139558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697468.139567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f5187389180 x1874531145867648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145867648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697468.139600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697468.139618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697468.139628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697468/real 1787697468] req@ffff8f518738a300 x1874531145867776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697468.139637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145867776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697468.139644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697468.139647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697468.139651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145867776:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697469.034484:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697470.058501:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697471.082535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697472.106547:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697473.130532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697473.258459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697473.258470:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258477:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258486:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258491:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697473.258523:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145867904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697473.258553:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258556:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697473.258558:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258563:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258566:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258570:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697473.258579:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697473.258593:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145868032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697473.258601:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258604:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697473.258605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145867904:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697473.258605:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258612:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258616:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258620:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697473.258625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697473.258639:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145868160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697473.258648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697473.258653:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258657:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258661:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258664:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697473.258668:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697473.258669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697473.258679:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145868288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697473.258686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697473.258689:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258693:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258695:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697473.258699:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697473.258706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145868416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697473.258711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f518738b800 x1874531145867904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697473.258714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258721:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697473.258724:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258732:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258735:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697473.258738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145867904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697473.258742:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697473.258750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697473.258762:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697473.258769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.258774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145867904:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697473.258779:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145868544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697473.258790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697473.258795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697473.258800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868032:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697473.258800:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00080000:3.0:1787697473.258805:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258810:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697473.258815:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697473.258826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697473.258829:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145868672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697473.258840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f518738b480 x1874531145868032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697473.258841:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697473.258853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697473.258860:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00100000:0.0:1787697473.258865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145868032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697473.258865:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258874:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697473.258882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697473.258882:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697473.258888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.258891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697473.258899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697473.258900:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145868800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697473.258914:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697473.258917:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697473.258920:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697473.258922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697473.258929:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5372 00000100:00100000:0.0:1787697473.258933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f5187388000 x1874531145868160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697473.258934:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697473.258942:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697473.258946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145868160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697473.258950:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697473.258952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.258956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.258958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697473.258963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697473.258966:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145868928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697473.258977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697473.258987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f518738a680 x1874531145868288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697473.258994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145868288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697473.258999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.259001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.259003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697473.259006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697473.259014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697473.259019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f5187388380 x1874531145868416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697473.259029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145868416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697473.259034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.259035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.259037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145868416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697473.259040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697473.259048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697473.259053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f518738b100 x1874531145868544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697473.259059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145868544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697473.259064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.259065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.259067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697473.259077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697473.259086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697473.259091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f5187388a80 x1874531145868672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697473.259098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145868672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697473.259106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.259107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.259109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697473.259112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697473.259120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697473.259125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f5187388e00 x1874531145868800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697473.259135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145868800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697473.259142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.259145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.259148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697473.259153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697473.259169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697473.259179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697473/real 1787697473] req@ffff8f5187389f80 x1874531145868928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697473.259189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145868928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697473.259195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697473.259198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697473.259201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145868928:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697474.154545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697475.178524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697476.202519:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697477.226543:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697478.251478:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697478.378494:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378504:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697478.378507:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378517:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378527:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697478.378533:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697478.378568:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145869056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697478.378599:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378603:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697478.378605:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378611:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378614:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697478.378619:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697478.378628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697478.378639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145869056:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697478.378653:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145869184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697478.378678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697478.378683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378686:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378690:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697478.378694:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697478.378698:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697478.378711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145869312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697478.378728:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378732:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697478.378734:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378737:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378741:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787697478.378747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697478.378748:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697478.378755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697478.378768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531145869440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697478.378788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace93100 x1874531145869056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697478.378790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378797:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697478.378801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378808:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378811:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697478.378814:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697478.378818:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697478.378828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145869056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697478.378834:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531145869568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697478.378838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.378846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697478.378852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697478.378852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145869056:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697478.378860:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697478.378863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378867:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378870:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697478.378874:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697478.378876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869184:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787697478.378881:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697478.378896:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531145869696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697478.378902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697478.378913:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378916:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697478.378918:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697478.378923:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.378927:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697478.378928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace90e00 x1874531145869184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697478.378934:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697478.378942:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697478.378957:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145869824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697478.378962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145869184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697478.378975:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.378981:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697478.378981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697478.378984:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697478.378986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697478.378993:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00100000:3.0:1787697478.378993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869184:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697478.379000:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697478.379003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869312:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697478.379008:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697478.379016:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697478.379024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697478.379031:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145869952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697478.379037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace92680 x1874531145869312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145869312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697478.379055:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697478.379062:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697478.379066:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697478.379073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697478.379075:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5378 00000100:00080000:0.0:1787697478.379081:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697478.379082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697478.379088:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869312:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787697478.379096:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697478.379100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697478.379125:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145870080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697478.379129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace90000 x1874531145869440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145869440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697478.379169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.379172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697478.379175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697478.379180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697478.379197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace93800 x1874531145869568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145869568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697478.379224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.379227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697478.379230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145869568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697478.379247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145869696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697478.379265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace93b80 x1874531145869696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145869696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697478.379285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.379287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697478.379289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145869696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697478.379292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145869824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697478.379302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace90380 x1874531145869824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145869824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697478.379318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.379319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697478.379321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145869824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697478.379324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145869952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697478.379332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace93100 x1874531145869952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145869952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697478.379347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.379348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697478.379350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145869952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697478.379369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145870080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697478.379380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697478.379387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697478/real 1787697478] req@ffff8f51ace90700 x1874531145870080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697478.379393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145870080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697478.379398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697478.379400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697478.379401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145870080:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697479.274517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697480.298456:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697481.322523:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697482.346577:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697483.370516:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697483.498511:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498524:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697483.498529:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498541:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498555:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498563:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697483.498624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145870208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498655:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498658:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697483.498660:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498665:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498669:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498673:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498686:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697483.498696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145870208:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697483.498700:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145870336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697483.498718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498722:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498725:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498728:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498733:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697483.498744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145870464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498752:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697483.498757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498761:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498764:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498768:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498773:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697483.498783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145870592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498795:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697483.498797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498801:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498804:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498808:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498812:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697483.498824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145870720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498833:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498836:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697483.498838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697483.498842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870336:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697483.498846:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498852:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498856:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498861:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697483.498874:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145870848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697483.498883:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697483.498884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498893:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697483.498895:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498899:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498903:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.498906:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697483.498918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697483.498925:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145870976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498936:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697483.498942:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498947:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498951:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697483.498955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870720:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697483.498958:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.498966:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697483.498979:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145871104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697483.498988:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697483.498991:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697483.498993:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697483.498996:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5383 00000100:00080000:3.0:1787697483.498999:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697483.499002:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697483.499007:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697483.499009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145870848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697483.499019:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145871232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697483.499037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145870976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697483.499052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145871104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697483.499071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145871232:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697484.394613:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697485.418588:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697486.442514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697486.570739:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787697486.570795:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697486.570815:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570873:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570901:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570911:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570921:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570930:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570938:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570946:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697486.570955:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787697486.573873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f518738b800 x1874531145871232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.573909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145871232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.573919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.573923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.573926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145871232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697486.573933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f518738a680 x1874531145870848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.573945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145870848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.573953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.573957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.573960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145870848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697486.573968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f5187388000 x1874531145870976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.573980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145870976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.573993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.573995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.573997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145870976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697486.574002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f518738b480 x1874531145871104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.574010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145871104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.574016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.574018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.574020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145871104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697486.574024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f5187388e00 x1874531145870336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.574030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145870336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.574035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.574039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.574042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697486.574049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f5187388a80 x1874531145870464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.574061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145870464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.574070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.574074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.574077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697486.574085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f518738b100 x1874531145870592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.574096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145870592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.574104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.574108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.574111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697486.574118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f5187388380 x1874531145870720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.574130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145870720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.574137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.574140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.574143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145870720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697486.574150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697483/real 1787697486] req@ffff8f5187389f80 x1874531145870208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697486.574160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145870208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697486.574165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697486.574168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697486.574170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145870208:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787697487.466598:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697488.490578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697488.618511:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697488.618525:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697488.618530:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.618544:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00080000:3.0:1787697488.618557:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618565:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697488.618609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145871360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697488.618643:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697488.618646:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697488.618648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.618654:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00080000:3.0:1787697488.618657:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618661:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697488.618671:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697488.618684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145871488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697488.618694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697488.618697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697488.618699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.618703:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00100000:0.0:1787697488.618709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145871360:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697488.618709:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618714:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697488.618724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697488.618740:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145871616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697488.618750:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697488.618753:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697488.618755:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.618759:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00080000:3.0:1787697488.618762:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618766:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697488.618771:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697488.618782:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145871744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697488.618789:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697488.618792:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697488.618796:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697488.618797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697488.618804:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00080000:3.0:1787697488.618809:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618813:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697488.618817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697488.618828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145871872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697488.618836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697488.618840:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697488.618842:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.618848:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00100000:0.0:1787697488.618853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f518738a300 x1874531145871360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697488.618854:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618863:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697488.618872:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697488.618881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145871360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697488.618888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145872000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697488.618896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697488.618901:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697488.618903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697488.618907:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697488.618912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145871360:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697488.618912:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.618922:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00100000:0.0:1787697488.618926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871488:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697488.618928:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.618936:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697488.618942:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697488.618955:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145872128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697488.618957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.618969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f5187389180 x1874531145871488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697488.618990:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697488.618996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145871488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697488.618997:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697488.619001:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.619009:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00080000:0.0:1787697488.619013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697488.619015:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697488.619020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697488.619024:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871488:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697488.619033:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697488.619048:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145872256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697488.619060:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697488.619061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871616:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697488.619067:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697488.619071:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697488.619075:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5388 00000100:00080000:3.0:1787697488.619079:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697488.619083:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697488.619087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697488.619089:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697488.619100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f518738aa00 x1874531145871616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697488.619104:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145872384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697488.619128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145871616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697488.619151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697488.619171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f5187389500 x1874531145871744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697488.619194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145871744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697488.619214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697488.619228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f518738ad80 x1874531145871872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697488.619248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145871872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145871872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697488.619265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697488.619278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f5187389c00 x1874531145872000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697488.619298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145872000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697488.619317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697488.619331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f518738bb80 x1874531145872128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697488.619349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145872128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697488.619421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697488.619438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f5187389f80 x1874531145872256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697488.619458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145872256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697488.619476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697488.619503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697488.619512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697488/real 1787697488] req@ffff8f5187388380 x1874531145872384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697488.619523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145872384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697488.619531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697488.619534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697488.619538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145872384:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697489.514568:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697490.538528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697491.562574:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697492.586513:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697493.610513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697493.738480:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697493.738490:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738498:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738506:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738511:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697493.738538:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145872512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697493.738558:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738561:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697493.738562:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738565:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738567:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738569:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738579:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697493.738587:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145872640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697493.738595:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738597:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697493.738598:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738602:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738604:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697493.738605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145872512:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697493.738608:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738615:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697493.738620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145872768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697493.738625:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738628:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697493.738629:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738631:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738632:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738634:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738637:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697493.738647:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145872896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697493.738652:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697493.738655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738658:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738660:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738663:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738665:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697493.738673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145873024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697493.738674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697493.738682:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697493.738686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738688:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738691:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738693:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697493.738701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145873152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697493.738707:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738709:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697493.738709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738713:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738715:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738719:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738722:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697493.738728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145873280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697493.738730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace91c00 x1874531145872512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697493.738734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697493.738742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697493.738747:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738751:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697493.738753:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738756:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697493.738763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145872512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697493.738765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145873408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697493.738779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697493.738780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697493.738785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697493.738788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697493.738788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697493.738795:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5393 00000100:00080000:0.0:1787697493.738799:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697493.738799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145872512:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697493.738805:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697493.738813:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697493.738820:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145873536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697493.738828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145872640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697493.738856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.738869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace91880 x1874531145872640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.738883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145872640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.738891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.738895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.738898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145872640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697493.738904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145872768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697493.738923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.738933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace91500 x1874531145872768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.738944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145872768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.738951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.738954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.738956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145872768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697493.738962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145872896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697493.738981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.738991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace92300 x1874531145872896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.739009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145872896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.739017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.739020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.739031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145872896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697493.739037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145873024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697493.739058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.739069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace93480 x1874531145873024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.739080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145873024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.739088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.739091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.739094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145873024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697493.739099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697493.739117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.739129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace92d80 x1874531145873152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.739140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145873152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.739153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.739156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.739159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697493.739165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697493.739185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.739196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace91180 x1874531145873280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.739206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145873280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.739214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.739216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.739219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697493.739224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697493.739242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.739253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace90700 x1874531145873408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.739264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145873408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.739271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.739274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.739278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697493.739299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697493.739317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697493.739327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697493/real 1787697493] req@ffff8f51ace93100 x1874531145873536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697493.739337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145873536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697493.739344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697493.739347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697493.739350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145873536:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697494.634565:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697495.658506:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697496.682549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697497.706538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697498.730825:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697498.858484:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697498.858504:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.858517:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858530:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.858539:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697498.858586:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145873664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697498.858614:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697498.858621:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.858626:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858630:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.858635:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697498.858646:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697498.858655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145873664:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697498.858664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145873792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697498.858677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858680:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697498.858682:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.858686:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858690:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.858694:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697498.858700:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697498.858713:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145873920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697498.858723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858727:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697498.858729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697498.858734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697498.858735:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858741:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.858745:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697498.858752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697498.858774:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145874048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697498.858784:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858788:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697498.858790:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.858794:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858797:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.858801:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697498.858806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697498.858819:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145874176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697498.858829:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858833:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697498.858835:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.858840:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858845:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697498.858846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183645c00 x1874531145873664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697498.858852:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697498.858862:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697498.858874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145873664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697498.858878:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145874304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697498.858898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.858904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697498.858910:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697498.858911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145873664:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697498.858917:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697498.858921:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.858926:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.858930:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.858942:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697498.858946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145873792:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697498.858950:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697498.858966:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145874432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697498.858977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697498.858985:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.858990:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697498.858992:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697498.858994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183645180 x1874531145873792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697498.858999:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.859007:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.859014:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145873792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697498.859021:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787697498.859031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697498.859039:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145874560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697498.859043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145873792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697498.859056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145873920:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697498.859057:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697498.859064:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697498.859066:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697498.859071:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5398 00000100:00080000:1.0:1787697498.859076:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697498.859081:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:3.0:1787697498.859085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697498.859089:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697498.859100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183647100 x1874531145873920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697498.859115:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145874688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697498.859126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145873920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145873920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697498.859155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145874048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697498.859177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183647800 x1874531145874048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697498.859201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145874048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145874048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697498.859220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145874176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697498.859239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183644e00 x1874531145874176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697498.859261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145874176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145874176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697498.859281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697498.859301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183647480 x1874531145874304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697498.859338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145874304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697498.859408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697498.859429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183646680 x1874531145874432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697498.859451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145874432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697498.859471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697498.859490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183644000 x1874531145874560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697498.859512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145874560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697498.859532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697498.859548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697498.859556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff8f5183644700 x1874531145874688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697498.859567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145874688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697498.859574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697498.859578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697498.859580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145874688:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697499.754541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697500.778564:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697501.802586:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697502.826586:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697503.850526:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697503.978419:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.978426:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697503.978429:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978437:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:3.0:1787697503.978445:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697503.978452:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697503.978487:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145874816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697503.978514:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.978518:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697503.978520:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978525:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:3.0:1787697503.978529:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697503.978533:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697503.978542:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697503.978556:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145874944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697503.978561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145874816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697503.978572:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.978576:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697503.978577:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978582:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:3.0:1787697503.978586:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697503.978590:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697503.978595:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697503.978608:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145875072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697503.978617:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.978620:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697503.978622:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978625:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:3.0:1787697503.978629:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697503.978633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697503.978636:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697503.978653:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697503.978666:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145875200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697503.978676:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697503.978681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f518738b100 x1874531145874816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.978682:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697503.978686:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697503.978702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145874816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.978703:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:3.0:1787697503.978710:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697503.978716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697503.978717:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697503.978723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697503.978725:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697503.978727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145874816:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697503.978741:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145875328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697503.978749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145874944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697503.978757:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.978762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787697503.978771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697503.978772:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978779:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00100000:0.0:1787697503.978783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f5187388a80 x1874531145874944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.978785:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697503.978793:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697503.978798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145874944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697503.978800:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697503.978811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697503.978815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145875456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697503.978817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697503.978822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145874944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697503.978833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145875072:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697503.978846:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697503.978850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697503.978857:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697503.978859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f5187388e00 x1874531145875072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.978860:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978870:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00100000:0.0:1787697503.978874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145875072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.978875:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697503.978881:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697503.978892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697503.978896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697503.978898:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697503.978901:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145875072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697503.978910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145875200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697503.978913:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145875584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697503.978923:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.978926:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697503.978928:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978932:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:3.0:1787697503.978936:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697503.978938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697503.978942:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697503.978948:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697503.978956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f518738b480 x1874531145875200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697503.978962:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145875712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697503.978976:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697503.978979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145875200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.978980:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697503.978984:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.978992:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5403 00000100:00080000:0.0:1787697503.978996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697503.978997:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697503.979001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697503.979005:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697503.979008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145875200:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787697503.979012:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697503.979026:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145875840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697503.979028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145875328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697503.979060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697503.979072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f5187388000 x1874531145875328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697503.979082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145875328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697503.979090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697503.979093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697503.979095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145875328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697503.979101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697503.979118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697503.979126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f518738a680 x1874531145875456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697503.979132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145875456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697503.979138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697503.979140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697503.979142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697503.979145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697503.979155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697503.979160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f518738b800 x1874531145875584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697503.979166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145875584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697503.979170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697503.979172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697503.979173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697503.979176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697503.979184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697503.979189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f518738b100 x1874531145875712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697503.979195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145875712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697503.979199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697503.979200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697503.979202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697503.979212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697503.979242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697503.979253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff8f5187388380 x1874531145875840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697503.979264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145875840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697503.979272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697503.979274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697503.979277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145875840:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697504.874514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697505.898502:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697506.922508:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697507.946512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697508.970542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697509.098463:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098473:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697509.098477:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098486:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098496:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098502:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697509.098535:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145875968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697509.098558:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098560:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697509.098561:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098565:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098567:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098569:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697509.098576:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697509.098584:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145876096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697509.098590:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098593:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697509.098595:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098597:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098598:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098600:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697509.098608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145875968:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697509.098610:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697509.098625:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145876224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697509.098635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098637:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697509.098639:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098642:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098645:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098648:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697509.098654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697509.098664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145876352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697509.098673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697509.098679:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098683:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697509.098685:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098690:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098693:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098696:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697509.098701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697509.098711:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145876480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697509.098721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f5187389f80 x1874531145875968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697509.098721:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697509.098732:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098741:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098746:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098752:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697509.098753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145875968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697509.098755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697509.098768:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145876608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697509.098769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.098777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.098783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145875968:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697509.098789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697509.098801:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697509.098805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697509.098808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697509.098809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697509.098817:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098823:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697509.098827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f518738bb80 x1874531145876096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697509.098831:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697509.098839:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697509.098847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145876096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697509.098855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145876736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697509.098859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.098868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697509.098873:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697509.098875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697509.098879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697509.098883:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098889:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00080000:3.0:1787697509.098893:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697509.098897:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697509.098902:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697509.098905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697509.098916:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145876864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697509.098928:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697509.098930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697509.098934:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697509.098937:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697509.098942:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5408 00000100:00100000:0.0:1787697509.098943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f5187389c00 x1874531145876224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697509.098948:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697509.098955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145876224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697509.098957:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697509.098966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697509.098967:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697509.098970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.098974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697509.098979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697509.098985:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145876992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697509.098994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697509.099004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f518738ad80 x1874531145876352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697509.099016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145876352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697509.099021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.099023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.099025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697509.099028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697509.099036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697509.099042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f5187389500 x1874531145876480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697509.099048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145876480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697509.099055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.099058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.099061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145876480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697509.099066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697509.099081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697509.099093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f518738aa00 x1874531145876608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697509.099102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145876608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697509.099112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.099113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.099116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697509.099119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697509.099131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697509.099154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f5187389180 x1874531145876736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697509.099165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145876736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697509.099173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.099176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.099179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697509.099194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697509.099209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697509.099219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f518738a300 x1874531145876864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697509.099230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145876864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697509.099237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.099240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.099243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697509.099249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697509.099264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697509.099271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697509/real 1787697509] req@ffff8f518738bb80 x1874531145876992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697509.099277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145876992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697509.099282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697509.099284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697509.099285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145876992:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697509.994518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697511.018518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697512.042529:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697513.066515:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697514.090536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697514.218459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697514.218470:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697514.218477:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218485:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218491:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697514.218525:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145877120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697514.218544:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218550:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697514.218552:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697514.218555:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218557:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218559:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218567:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697514.218575:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145877248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697514.218582:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218584:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697514.218587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697514.218589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145877120:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697514.218590:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218594:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218598:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218601:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697514.218609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145877376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697514.218620:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697514.218623:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697514.218629:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218631:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218634:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218637:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697514.218643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145877504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697514.218648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697514.218653:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697514.218656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697514.218657:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218661:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218663:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218665:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697514.218672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145877632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697514.218678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697514.218681:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697514.218683:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218686:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218688:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697514.218696:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145877760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697514.218702:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218706:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697514.218707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f5187389f80 x1874531145877120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697514.218708:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697514.218715:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218720:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218723:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218726:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697514.218730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145877120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697514.218733:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145877888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697514.218747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697514.218748:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697514.218754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697514.218754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697514.218757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697514.218760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145877120:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697514.218764:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218767:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218769:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218772:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697514.218777:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145878016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697514.218784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697514.218785:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697514.218793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697514.218795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697514.218798:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5413 00000100:00080000:3.0:1787697514.218800:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697514.218802:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697514.218806:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697514.218812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697514.218815:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145878144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697514.218825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f5187388380 x1874531145877248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.218833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145877248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.218839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.218843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.218846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697514.218851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697514.218870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.218880:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f518738b100 x1874531145877376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.218896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145877376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.218904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.218907:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.218911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697514.218916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697514.218935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.218945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f518738b800 x1874531145877504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.218959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145877504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.218964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.218966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.218968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697514.218971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697514.218984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.218990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f518738a680 x1874531145877632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.218996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145877632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.219001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.219002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.219004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145877632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697514.219007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145877760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697514.219015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.219020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f5187388000 x1874531145877760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.219026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145877760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.219035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.219037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.219038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145877760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697514.219041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145877888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697514.219077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.219084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f518738b480 x1874531145877888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.219090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145877888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.219095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.219096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.219098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145877888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697514.219110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145878016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697514.219120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.219128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f5187388e00 x1874531145878016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.219138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145878016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.219145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.219148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.219151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145878016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697514.219157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145878144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697514.219174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697514.219185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697514/real 1787697514] req@ffff8f5187388a80 x1874531145878144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697514.219196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145878144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697514.219203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697514.219206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697514.219209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145878144:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697515.114541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697516.138525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697517.162500:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697518.186520:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697519.210500:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697519.338457:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338465:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697519.338468:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338475:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338483:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338488:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697519.338522:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145878272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697519.338549:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697519.338553:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338556:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338561:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338568:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697519.338577:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145878400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697519.338583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697519.338587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338589:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338591:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338593:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697519.338602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145878272:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697519.338603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145878528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697519.338614:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697519.338617:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338619:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338621:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338623:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697519.338632:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145878656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697519.338637:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338639:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697519.338640:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338643:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338645:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338647:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338650:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697519.338655:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145878784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697519.338660:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338662:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697519.338664:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697519.338668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697519.338670:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338674:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338676:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338679:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697519.338686:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145878912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697519.338691:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697519.338694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338696:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338699:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338701:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697519.338706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f5187388a80 x1874531145878272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697519.338712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145879040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697519.338725:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145878272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697519.338727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338733:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697519.338736:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338743:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:0.0:1787697519.338744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697519.338745:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697519.338751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697519.338752:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697519.338761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145878272:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697519.338768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145879168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697519.338770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878400:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697519.338785:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697519.338787:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697519.338789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697519.338792:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5418 00000100:00080000:3.0:1787697519.338794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697519.338797:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697519.338801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697519.338802:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697519.338811:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145879296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697519.338813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f5187388e00 x1874531145878400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.338833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145878400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.338838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.338841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.338843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697519.338859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697519.338871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.338877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f518738b480 x1874531145878528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.338884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145878528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.338889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.338890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.338892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697519.338900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697519.338910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.338916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f5187388000 x1874531145878656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.338936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145878656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.338943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.338946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.338949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697519.338980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697519.338998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.339005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f518738a680 x1874531145878784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.339012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145878784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.339017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.339019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.339021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145878784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697519.339025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145878912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697519.339034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.339039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f518738b800 x1874531145878912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.339046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145878912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.339059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.339063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.339066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145878912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697519.339077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145879040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697519.339096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.339104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f518738b100 x1874531145879040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.339110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145879040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.339115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.339117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.339119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145879040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697519.339122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145879168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697519.339133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.339138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f5187388380 x1874531145879168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.339144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145879168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.339148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.339150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.339152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145879168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697519.339155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145879296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697519.339163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697519.339168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697519/real 1787697519] req@ffff8f5187389f80 x1874531145879296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697519.339174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145879296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697519.339178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697519.339180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697519.339181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145879296:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697520.234589:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697521.258494:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697522.282534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697523.306522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697524.330525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697524.458491:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458500:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697524.458505:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458514:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458525:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458533:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697524.458566:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145879424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697524.458593:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458597:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697524.458599:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458604:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458608:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697524.458621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697524.458633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145879424:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697524.458635:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531145879552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697524.458654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697524.458658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458662:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458665:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458669:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697524.458674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697524.458686:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531145879680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697524.458694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697524.458699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458704:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5424 00000800:00000100:3.0:1787697524.458705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697524.458707:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458712:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697524.458716:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697524.458726:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531145879808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697524.458735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458738:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697524.458740:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458743:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458747:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458752:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697524.458752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace90380 x1874531145879424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697524.458759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697524.458772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145879936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697524.458779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145879424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697524.458781:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697524.458788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458795:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458800:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697524.458801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697524.458803:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697524.458808:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697524.458809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697524.458817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145880064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697524.458818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145879424:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697524.458830:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697524.458832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697524.458833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458837:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458840:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458844:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697524.458849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879552:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787697524.458850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697524.458878:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145880192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697524.458889:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787697524.458894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697524.458896:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697524.458900:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458905:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00080000:0.0:1787697524.458909:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787697524.458914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace93b80 x1874531145879552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697524.458915:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697524.458923:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697524.458933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145879552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697524.458938:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145880320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697524.458949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697524.458955:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697524.458956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697524.458962:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787697524.458963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697524.458966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697524.458972:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5424 00000100:00100000:3.0:1787697524.458973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879680:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697524.458977:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697524.458983:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697524.458987:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697524.458997:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145880448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787697524.459003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace93800 x1874531145879680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145879680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697524.459058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697524.459079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace90000 x1874531145879808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145879808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697524.459121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697524.459141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace92680 x1874531145879936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145879936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145879936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697524.459189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697524.459208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace90e00 x1874531145880064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145880064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697524.459262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697524.459282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace90380 x1874531145880192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145880192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697524.459323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697524.459341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace93100 x1874531145880320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145880320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697524.459401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697524.459415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697524.459424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697524/real 1787697524] req@ffff8f51ace90700 x1874531145880448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697524.459434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145880448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697524.459441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697524.459443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697524.459445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145880448:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697525.354506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697526.378509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697527.402510:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697528.426565:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697529.450538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697529.578441:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697529.578459:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578472:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578490:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578498:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697529.578538:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145880576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578569:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578573:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697529.578574:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578579:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578582:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578586:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697529.578605:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145880704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578612:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578618:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697529.578620:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578623:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578627:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697529.578628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145880576:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697529.578630:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578638:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697529.578650:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145880832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578660:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697529.578665:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578673:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578677:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578682:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697529.578694:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145880960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578706:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697529.578710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697529.578711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697529.578718:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578726:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697529.578742:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145881088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578750:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697529.578756:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578762:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00100000:0.0:1787697529.578763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f518738bb80 x1874531145880576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697529.578766:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578774:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578794:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697529.578806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145880576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697529.578809:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145881216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697529.578834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697529.578834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578841:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697529.578842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697529.578843:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697529.578849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145880576:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697529.578853:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578859:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697529.578865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145880704:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697529.578866:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697529.578888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145881344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578898:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578901:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697529.578903:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578907:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5429 00000800:00000100:0.0:1787697529.578912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697529.578912:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578918:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697529.578923:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697529.578931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f518738a300 x1874531145880704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697529.578936:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145881472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697529.578959:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697529.578963:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697529.578965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145880704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697529.578966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697529.578973:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5429 00000100:00080000:3.0:1787697529.578978:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697529.578983:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697529.578986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697529.578988:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697529.578992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.578996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145880704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697529.579001:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145881600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697529.579025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145880832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697529.579048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f5187389180 x1874531145880832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145880832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145880832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697529.579093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145880960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697529.579111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f518738aa00 x1874531145880960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145880960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145880960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697529.579155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145881088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697529.579173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f5187389500 x1874531145881088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145881088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145881088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697529.579217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697529.579231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f518738ad80 x1874531145881216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145881216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697529.579256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697529.579265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f5187389c00 x1874531145881344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145881344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697529.579291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697529.579301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f5187389f80 x1874531145881472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145881472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697529.579330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697529.579341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697529.579347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697529/real 1787697529] req@ffff8f5187388380 x1874531145881600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697529.579371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145881600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697529.579376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697529.579378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697529.579380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145881600:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697530.474531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697531.498552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697532.522517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697533.546569:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697534.570468:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697534.698431:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698438:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697534.698442:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698449:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698457:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698462:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697534.698488:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145881728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697534.698509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698512:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697534.698513:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698516:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698519:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698521:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698527:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697534.698534:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145881856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697534.698541:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698543:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697534.698544:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698546:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698548:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698551:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698555:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697534.698565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145881728:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697534.698570:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145881984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697534.698581:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697534.698586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698589:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698592:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698595:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698600:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697534.698609:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145882112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697534.698620:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698622:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697534.698623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698626:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698629:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698631:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698633:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697534.698640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697534.698642:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145882240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697534.698649:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697534.698653:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698655:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698657:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698660:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698662:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697534.698668:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145882368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697534.698673:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698675:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697534.698677:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698680:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698683:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698687:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697534.698689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f518738b100 x1874531145881728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697534.698703:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697534.698716:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145882496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697534.698724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145881728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697534.698724:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697534.698733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697534.698742:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698745:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698747:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698749:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697534.698753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697534.698758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145882624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697534.698761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.698769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145881728:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697534.698771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697534.698776:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697534.698778:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697534.698782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145881856:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697534.698783:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5434 00000100:00080000:3.0:1787697534.698787:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697534.698790:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697534.698793:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697534.698801:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145882752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697534.698811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.698824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f518738b800 x1874531145881856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.698832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145881856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.698837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.698839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.698841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145881856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697534.698858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145881984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697534.698870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.698876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f518738a680 x1874531145881984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.698883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145881984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.698887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.698889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.698890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145881984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697534.698893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145882112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697534.698902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.698907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f5187388000 x1874531145882112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.698914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145882112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.698918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.698919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.698921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145882112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697534.698924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145882240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697534.698932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.698937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f518738b480 x1874531145882240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.698947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145882240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.698951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.698953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.698954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145882240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697534.698957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697534.698965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.698970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f5187388e00 x1874531145882368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.698976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145882368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.698980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.698982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.698983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697534.698986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697534.698995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.699000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f5187388a80 x1874531145882496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.699006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145882496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.699014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.699015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.699017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697534.699020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697534.699028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.699033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f5187388380 x1874531145882624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.699038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145882624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.699042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.699044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.699046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697534.699048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697534.699056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697534.699060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697534/real 1787697534] req@ffff8f5187389f80 x1874531145882752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697534.699066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145882752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697534.699070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697534.699072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697534.699074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145882752:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697535.594520:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697536.618505:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697537.642577:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697538.666530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697539.690524:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697539.818409:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818418:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697539.818421:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818429:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818438:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818443:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697539.818472:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145882880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697539.818492:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818494:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697539.818496:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818499:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818501:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818503:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818509:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697539.818517:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145883008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697539.818523:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818525:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697539.818528:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818530:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818532:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818534:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818537:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697539.818544:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145883136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697539.818550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145882880:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697539.818552:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697539.818561:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818565:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818566:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818568:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818571:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697539.818578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145883264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697539.818583:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818585:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697539.818587:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818589:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818591:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818594:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818598:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697539.818629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697539.818638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145883392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697539.818648:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818651:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697539.818653:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818657:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818661:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697539.818667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f5187389c00 x1874531145882880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697539.818668:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697539.818686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145883520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697539.818693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697539.818694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145882880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697539.818697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697539.818699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818706:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818710:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818716:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697539.818720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697539.818720:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697539.818728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697539.818731:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145883648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697539.818735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145882880:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697539.818743:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818746:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697539.818748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883008:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697539.818749:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818757:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818759:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697539.818761:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818763:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697539.818771:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145883776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697539.818778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697539.818781:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697539.818784:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697539.818785:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697539.818788:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5439 00000100:00080000:3.0:1787697539.818792:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697539.818793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f518738ad80 x1874531145883008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697539.818798:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697539.818804:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697539.818811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145883008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697539.818815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145883904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697539.818832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.818836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.818838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697539.818858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697539.818872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.818878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f5187389500 x1874531145883136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.818885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145883136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.818891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.818894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.818897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697539.818903:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697539.818920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.818931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f518738aa00 x1874531145883264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.818942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145883264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.818949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.818952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.818955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697539.818961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697539.818979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.819003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f5187389180 x1874531145883392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.819020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145883392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.819027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.819030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.819033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145883392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697539.819039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697539.819056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.819063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f518738a300 x1874531145883520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.819070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145883520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.819076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.819077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.819079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697539.819084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697539.819100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.819110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f518738bb80 x1874531145883648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.819120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145883648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.819133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.819135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.819138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697539.819143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697539.819162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.819172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f5187389f80 x1874531145883776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.819183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145883776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.819189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.819191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.819193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697539.819199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697539.819218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697539.819228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697539/real 1787697539] req@ffff8f5187388380 x1874531145883904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697539.819239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145883904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697539.819245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697539.819247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697539.819249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145883904:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697540.714511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697541.738523:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697542.762483:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697543.786509:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697544.810547:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697544.938468:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938475:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697544.938478:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938486:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938494:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938499:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697544.938556:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145884032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697544.938585:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938589:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697544.938591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938595:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938604:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938612:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697544.938626:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145884160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697544.938631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145884032:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697544.938636:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697544.938642:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938645:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938648:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938650:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938653:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697544.938660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145884288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697544.938666:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938668:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697544.938669:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938677:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697544.938685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145884416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697544.938690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697544.938695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787697544.938696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697544.938699:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938702:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938707:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697544.938714:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145884544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697544.938723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938725:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697544.938726:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938729:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938731:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938734:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697544.938741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace91180 x1874531145884032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697544.938745:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145884672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697544.938770:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697544.938770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145884032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697544.938774:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697544.938777:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938781:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938783:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938785:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938789:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787697544.938801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697544.938803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145884800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697544.938809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697544.938816:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697544.938817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145884032:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697544.938824:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697544.938826:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938831:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938835:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938838:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938843:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697544.938847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697544.938855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145884928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697544.938864:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697544.938866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697544.938868:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697544.938870:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5444 00000100:00080000:0.0:1787697544.938874:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697544.938876:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697544.938880:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787697544.938893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697544.938894:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145885056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697544.938913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace92d80 x1874531145884160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.938927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145884160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.938936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.938940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.938943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697544.938949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697544.938966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.938977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace93480 x1874531145884288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.938988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145884288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.938996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.938999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697544.939008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697544.939026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.939037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace92300 x1874531145884416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.939048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145884416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.939055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.939058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697544.939067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697544.939083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.939092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace91500 x1874531145884544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.939102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145884544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.939109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.939112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145884544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697544.939121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145884672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697544.939138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.939148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace91880 x1874531145884672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.939160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145884672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.939168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.939171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145884672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697544.939180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145884800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787697544.939197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.939207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace91c00 x1874531145884800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.939219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145884800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.939227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.939229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145884800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697544.939251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145884928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697544.939269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.939279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace91180 x1874531145884928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.939290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145884928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.939303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.939306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145884928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697544.939315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145885056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787697544.939333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697544.939343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff8f51ace90700 x1874531145885056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697544.939373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145885056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697544.939383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697544.939386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697544.939389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145885056:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697545.834528:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697546.858512:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697547.882555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697548.906542:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697549.930566:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697550.058441:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058451:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697550.058455:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058463:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058471:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058476:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697550.058503:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145885184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697550.058527:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058531:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697550.058532:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058537:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058540:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058544:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058553:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697550.058568:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145885312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697550.058569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145885184:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697550.058582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058585:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697550.058586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058589:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058591:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058593:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058596:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697550.058604:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145885440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697550.058610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058612:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697550.058613:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058615:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058617:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058619:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697550.058628:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145885568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697550.058634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058636:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697550.058637:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058639:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058640:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058642:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058645:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697550.058651:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531145885696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697550.058657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885312:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697550.058659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058663:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697550.058664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058667:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058669:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058672:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058675:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697550.058681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697550.058686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531145885824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697550.058694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697550.058699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058702:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00100000:3.0:1787697550.058704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885568:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787697550.058705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058710:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697550.058729:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531145885952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697550.058737:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697550.058741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697550.058743:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787697550.058747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885696:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787697550.058750:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058766:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697550.058781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145885824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697550.058792:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145886080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697550.058803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787697550.058808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145885952:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697550.058809:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697550.058812:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697550.058820:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5449 00000100:00080000:0.0:1787697550.058824:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697550.058828:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697550.058833:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697550.058843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145886080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697550.058848:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145886208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787697550.058888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145886208:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697550.954531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697551.978561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697553.002557:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697553.130665:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787697553.130705:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697553.130720:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697553.130768:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787697553.130782:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697553.130793:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697553.130803:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:3.0:1787697553.130804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace91180 x1874531145886208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697553.130815:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697553.130827:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:3.0:1787697553.130837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145886208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697553.130842:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697553.130853:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:3.0:1787697553.130856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.130863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.130868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145886208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697553.130876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace90700 x1874531145886080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.130889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145886080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.130896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.130899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.130903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145886080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697553.130908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace90000 x1874531145885696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.130918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145885696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.130924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.130927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.130931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697553.130937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace93800 x1874531145885824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.130955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145885824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.130963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.130966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.130969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145885824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697553.130976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace93b80 x1874531145885952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.130985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145885952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.130992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.130995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.130998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145885952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697553.131004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace90380 x1874531145885312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.131016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145885312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.131023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.131026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.131030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697553.131037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace90e00 x1874531145885440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.131047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145885440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.131054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.131057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.131060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697553.131068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace92680 x1874531145885568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.131079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145885568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.131087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.131090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.131093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145885568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697553.131099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697550/real 1787697553] req@ffff8f51ace93100 x1874531145885184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697553.131112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145885184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697553.131119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697553.131122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697553.131125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145885184:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787697554.026544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697555.050599:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697555.178486:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178499:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697555.178505:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.178518:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178531:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178540:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697555.178583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145886336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.178622:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178627:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697555.178629:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.178634:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178638:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178642:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.178655:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697555.178669:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145886464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.178679:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178683:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697555.178685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.178688:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178692:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178697:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.178702:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697555.178705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145886336:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697555.178716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145886592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.178727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178731:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697555.178733:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.178737:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178741:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178745:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.178752:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697555.178765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145886720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.178774:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178777:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697555.178780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.178784:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178789:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178793:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.178798:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697555.178804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697555.178812:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145886848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.178824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697555.178830:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.178835:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178839:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178843:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.178849:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697555.178862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f5187388a80 x1874531145886336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697555.178862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145886976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.178891:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178895:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697555.178897:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697555.178903:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145886336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697555.178904:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178911:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178917:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.178922:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697555.178935:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145887104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697555.178942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697555.178945:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697555.178950:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697555.178951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697555.178953:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697555.178960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145886336:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697555.178961:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.178964:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697555.178971:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697555.178978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886464:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697555.178978:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697555.178998:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145887232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697555.179009:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697555.179013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697555.179015:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697555.179019:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697555.179025:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5454 00000100:00080000:3.0:1787697555.179030:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697555.179034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f5187388e00 x1874531145886464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697555.179038:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697555.179046:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697555.179055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145886464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697555.179062:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145887360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697555.179068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697555.179113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697555.179145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f518738b480 x1874531145886592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145886592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697555.179192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697555.179214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f5187388000 x1874531145886720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145886720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697555.179247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697555.179265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f518738a680 x1874531145886848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145886848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145886848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697555.179310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145886976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697555.179329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f518738b800 x1874531145886976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145886976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145886976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697555.179428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145887104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697555.179450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f518738b100 x1874531145887104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145887104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145887104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697555.179495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145887232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697555.179515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f5187388380 x1874531145887232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145887232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145887232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697555.179548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145887360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697555.179561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697555.179567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697555/real 1787697555] req@ffff8f5187389f80 x1874531145887360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697555.179574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145887360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697555.179578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697555.179580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697555.179582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145887360:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697556.074520:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697557.098535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697558.122526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697559.146593:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697560.170569:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697560.298513:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298525:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697560.298530:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.298542:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.298556:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697560.298564:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697560.298611:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145887488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.298648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298652:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697560.298654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.298658:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.298661:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697560.298665:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697560.298676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697560.298690:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145887616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.298701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298706:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697560.298707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145887488:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697560.298709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.298718:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.298722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697560.298725:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697560.298729:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697560.298745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145887744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.298755:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697560.298760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.298764:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.298768:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697560.298772:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697560.298778:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697560.298790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145887872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697560.298794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697560.298805:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298808:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697560.298810:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.298816:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.298821:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697560.298825:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697560.298829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697560.298858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f518738bb80 x1874531145887488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697560.298858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145888000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.298885:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298889:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697560.298891:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697560.298895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145887488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697560.298899:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:0.0:1787697560.298913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697560.298914:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697560.298920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697560.298923:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697560.298929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145887488:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697560.298930:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697560.298947:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145888128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.298958:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.298961:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697560.298964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145887616:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697560.298964:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.298972:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.298976:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697560.298980:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697560.298986:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697560.298995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697560.298999:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145888256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.299012:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697560.299014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f518738a300 x1874531145887616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697560.299018:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697560.299022:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697560.299030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145887616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697560.299031:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.299038:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697560.299043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697560.299044:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697560.299050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697560.299052:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697560.299057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145887616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697560.299068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145887744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697560.299068:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145888384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697560.299087:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697560.299090:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697560.299092:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697560.299097:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5459 00000100:00080000:3.0:1787697560.299101:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697560.299105:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697560.299110:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697560.299117:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697560.299122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f5187389180 x1874531145887744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697560.299131:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145888512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697560.299139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145887744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145887744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697560.299164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145887872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697560.299185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697560.299196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f518738aa00 x1874531145887872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697560.299204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145887872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145887872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697560.299217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145888000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697560.299229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697560.299235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f5187389500 x1874531145888000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697560.299246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145888000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145888000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697560.299295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697560.299324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697560.299332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f518738ad80 x1874531145888128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697560.299340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145888128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697560.299418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697560.299450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697560.299459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f5187389c00 x1874531145888256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697560.299466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145888256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697560.299490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697560.299502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697560.299508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f518738bb80 x1874531145888384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697560.299515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145888384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697560.299534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697560.299551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697560.299561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697560/real 1787697560] req@ffff8f5187389f80 x1874531145888512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697560.299571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145888512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697560.299576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697560.299578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697560.299581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145888512:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697561.194655:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697562.218589:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697563.243155:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697564.266596:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697565.291204:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697565.418502:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418514:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697565.418519:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.418530:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418544:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418552:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697565.418598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145888640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697565.418636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697565.418642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.418648:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418652:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418656:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697565.418680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145888768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697565.418690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697565.418695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697565.418700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145888640:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697565.418701:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418706:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418712:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418717:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697565.418731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145888896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697565.418740:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418743:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697565.418745:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.418748:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418753:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418757:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418762:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697565.418772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145889024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697565.418781:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697565.418783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697565.418786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697565.418789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.418794:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418798:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418802:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418806:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697565.418817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145889152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697565.418826:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418830:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697565.418832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.418839:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00100000:0.0:1787697565.418840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f5187388380 x1874531145888640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697565.418843:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418850:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418858:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697565.418871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145888640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697565.418874:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145889280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697565.418896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697565.418897:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418904:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697565.418906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697565.418907:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697565.418914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145888640:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697565.418915:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418921:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697565.418927:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418933:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697565.418944:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145889408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697565.418948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145888768:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697565.418960:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697565.418964:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697565.418966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.418971:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:3.0:1787697565.418976:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697565.418980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697565.418983:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697565.418990:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697565.418998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f518738b100 x1874531145888768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697565.419004:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145889536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697565.419019:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697565.419021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145888768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697565.419023:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697565.419027:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697565.419036:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5465 00000100:00080000:0.0:1787697565.419040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697565.419042:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697565.419050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697565.419051:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145888768:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697565.419059:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697565.419071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145888896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697565.419076:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145889664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697565.419097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f518738b800 x1874531145888896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145888896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145888896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697565.419143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145889024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697565.419172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f518738a680 x1874531145889024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145889024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145889024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697565.419211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145889152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697565.419227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f5187388000 x1874531145889152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145889152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145889152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697565.419270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697565.419287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f518738b480 x1874531145889280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145889280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697565.419338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697565.419380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f5187388e00 x1874531145889408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145889408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697565.419425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697565.419444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f5187388a80 x1874531145889536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145889536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697565.419485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697565.419503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697565.419524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697565/real 1787697565] req@ffff8f5187388380 x1874531145889664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697565.419536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145889664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697565.419543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697565.419546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697565.419550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145889664:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697566.314539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697567.338629:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697568.362605:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697569.386512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697570.410564:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697570.538455:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538470:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697570.538475:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538490:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.538503:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697570.538512:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697570.538560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145889792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697570.538596:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538600:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697570.538602:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538607:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.538611:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697570.538616:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.538627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697570.538643:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145889920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697570.538655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145889792:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697570.538655:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697570.538668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538672:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.538675:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697570.538679:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.538685:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697570.538701:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145890048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697570.538711:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538714:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697570.538716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538720:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.538724:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697570.538728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697570.538732:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.538739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697570.538752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145890176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697570.538761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697570.538766:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538777:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.538781:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697570.538785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f5187389f80 x1874531145889792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697570.538788:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.538796:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697570.538813:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145890304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697570.538822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145889792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697570.538822:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697570.538831:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538841:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:0.0:1787697570.538845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697570.538845:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697570.538853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697570.538854:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697570.538859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145889792:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697570.538862:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697570.538876:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145890432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697570.538885:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.538889:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697570.538891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145889920:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697570.538903:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.538912:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.538916:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697570.538919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697570.538924:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.538930:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697570.538936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f518738bb80 x1874531145889920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697570.538946:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145890560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697570.538950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145889920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.538972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.538976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697570.538976:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697570.538979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145889920:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697570.538985:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697570.538989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145890048:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697570.538989:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.539000:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.539005:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697570.539009:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.539014:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697570.539026:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145890688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697570.539034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697570.539041:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697570.539044:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697570.539046:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697570.539051:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5470 00000100:00080000:3.0:1787697570.539054:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697570.539055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f5187389c00 x1874531145890048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697570.539060:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697570.539084:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697570.539085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145890048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697570.539102:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145890816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697570.539110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145890048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697570.539127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145890176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697570.539149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697570.539161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f518738ad80 x1874531145890176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697570.539171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145890176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.539182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145890176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697570.539190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145890304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697570.539221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697570.539231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f5187389500 x1874531145890304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697570.539243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145890304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.539250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145890304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697570.539255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697570.539267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697570.539272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f518738aa00 x1874531145890432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697570.539279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145890432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.539286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697570.539300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697570.539309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697570.539315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f5187389180 x1874531145890560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697570.539321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145890560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.539332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697570.539336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697570.539345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697570.539350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f518738a300 x1874531145890688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697570.539380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145890688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.539388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697570.539395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697570.539407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697570.539414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697570/real 1787697570] req@ffff8f5187389f80 x1874531145890816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697570.539421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145890816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697570.539425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697570.539427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697570.539429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145890816:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697571.434519:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697572.458532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697573.482580:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697574.506586:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697575.530566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697575.658466:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658474:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697575.658477:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697575.658485:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658496:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658502:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697575.658531:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145890944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697575.658555:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697575.658560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697575.658563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658566:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658570:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658577:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697575.658590:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145891072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697575.658600:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697575.658605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145890944:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697575.658605:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697575.658607:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697575.658616:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658619:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658622:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658628:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697575.658648:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145891200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697575.658657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658659:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697575.658661:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697575.658665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697575.658668:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658673:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658676:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697575.658691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145891328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697575.658700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697575.658704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697575.658705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f5187388380 x1874531145890944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697575.658709:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658714:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658719:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658722:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697575.658727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145890944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697575.658732:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145891456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697575.658738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697575.658741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697575.658744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697575.658745:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697575.658747:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697575.658749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145890944:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697575.658752:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658755:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658758:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658761:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697575.658767:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145891584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697575.658772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891072:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697575.658778:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658781:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697575.658782:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697575.658785:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658788:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658790:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658793:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697575.658796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697575.658802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145891712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697575.658809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f5187388a80 x1874531145891072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697575.658810:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658817:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697575.658820:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697575.658824:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00100000:1.0:1787697575.658827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145891072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697575.658828:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697575.658834:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697575.658838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697575.658840:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697575.658845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.658850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697575.658854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145891840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697575.658860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891200:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697575.658871:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697575.658875:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697575.658877:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697575.658881:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5475 00000100:00080000:3.0:1787697575.658886:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697575.658887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697575.658893:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697575.658899:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697575.658903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f5187388e00 x1874531145891200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697575.658912:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145891968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697575.658920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145891200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.658930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.658934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.658937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697575.658944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697575.658964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697575.658975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f518738b480 x1874531145891328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697575.659000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145891328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.659008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.659012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.659014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697575.659019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697575.659033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697575.659040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f5187388000 x1874531145891456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697575.659046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145891456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.659051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.659053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.659055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145891456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697575.659069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697575.659085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697575.659095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f518738a680 x1874531145891584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697575.659106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145891584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.659115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.659117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.659121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697575.659126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697575.659144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697575.659155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f518738b800 x1874531145891712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697575.659166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145891712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.659173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.659176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.659179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697575.659184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697575.659214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697575.659222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f518738b100 x1874531145891840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697575.659230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145891840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.659237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.659240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.659243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697575.659248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697575.659265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697575.659276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697575/real 1787697575] req@ffff8f5187388380 x1874531145891968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697575.659287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145891968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697575.659295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697575.659298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697575.659301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145891968:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697576.554580:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697577.578576:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697578.602582:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697579.626610:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697580.650632:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697580.778500:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778513:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697580.778518:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.778536:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.778551:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697580.778560:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697580.778611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145892096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697580.778641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697580.778647:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.778652:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.778655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697580.778660:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.778671:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697580.778686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145892096:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697580.778687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145892224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697580.778703:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697580.778708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.778713:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.778717:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697580.778720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.778726:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697580.778741:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145892352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697580.778750:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778755:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787697580.778757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697580.778758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.778765:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.778770:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697580.778774:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.778780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697580.778792:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145892480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697580.778801:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697580.778806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.778810:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00100000:1.0:1787697580.778821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f5187389f80 x1874531145892096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697580.778824:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697580.778833:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.778839:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697580.778850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145892096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697580.778854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145892608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697580.778875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697580.778876:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778883:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697580.778885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697580.778886:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697580.778891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145892096:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697580.778906:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.778911:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697580.778916:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.778920:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697580.778929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697580.778934:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145892736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697580.778946:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.778950:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787697580.778953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697580.778954:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.778959:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.778964:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697580.778975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f518738a300 x1874531145892224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697580.778976:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.778985:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697580.778990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145892224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697580.779001:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145892864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697580.779002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697580.779029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697580.779030:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697580.779036:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697580.779038:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697580.779044:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5480 00000800:00000100:1.0:1787697580.779047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697580.779053:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697580.779057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f5187389180 x1874531145892352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697580.779061:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697580.779072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145892352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697580.779073:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697580.779084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697580.779093:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145892992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697580.779095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892480:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697580.779108:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697580.779113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697580.779117:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697580.779120:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697580.779122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f518738aa00 x1874531145892480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697580.779127:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5480 00000100:00080000:3.0:1787697580.779136:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697580.779140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145892480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697580.779144:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697580.779153:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697580.779155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697580.779170:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145893120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697580.779176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697580.779196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697580.779206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f5187389500 x1874531145892608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697580.779213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145892608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697580.779219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145892608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697580.779234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145892736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697580.779244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697580.779250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f518738ad80 x1874531145892736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697580.779257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145892736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697580.779262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145892736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697580.779268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145892864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697580.779276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697580.779282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f5187389c00 x1874531145892864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697580.779289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145892864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697580.779293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145892864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697580.779299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145892992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697580.779307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697580.779312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f518738bb80 x1874531145892992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697580.779319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145892992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697580.779323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145892992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697580.779329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145893120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697580.779336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697580.779341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697580/real 1787697580] req@ffff8f5187389f80 x1874531145893120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697580.779348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145893120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697580.779380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697580.779383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697580.779386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145893120:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697581.674585:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697582.698569:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697583.722468:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697584.746540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697585.770512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697585.898505:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898515:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697585.898518:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898526:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898535:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898540:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697585.898568:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145893248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898591:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697585.898595:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898599:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898609:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697585.898616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145893376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898623:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898625:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697585.898626:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898629:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898630:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898632:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898635:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697585.898643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145893504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697585.898645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145893248:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697585.898650:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697585.898655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898658:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898659:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898661:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898664:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697585.898671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145893632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898676:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898678:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697585.898679:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898682:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898684:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898685:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898688:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697585.898695:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145893760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898703:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697585.898704:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898706:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898708:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898711:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697585.898715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898716:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697585.898724:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145893888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898732:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697585.898733:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898735:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898738:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898740:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898743:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697585.898748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145894016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697585.898757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697585.898761:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00100000:0.0:1787697585.898764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183644380 x1874531145893248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697585.898764:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898769:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898773:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697585.898781:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145894144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697585.898787:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697585.898790:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697585.898792:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697585.898793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145893248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697585.898796:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5485 00000100:00080000:1.0:1787697585.898799:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697585.898803:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697585.898820:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697585.898831:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145894272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697585.898846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.898853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.898860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145893248:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697585.898868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697585.898897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.898911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183647b80 x1874531145893376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.898923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145893376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.898931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.898935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.898938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697585.898960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697585.898978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.898989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183646a00 x1874531145893504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145893504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697585.899019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697585.899036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.899047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183646d80 x1874531145893632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145893632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697585.899076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697585.899091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.899099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183644700 x1874531145893760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145893760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145893760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697585.899135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145893888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697585.899152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.899160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183644000 x1874531145893888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145893888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145893888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697585.899178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145894016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697585.899187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.899192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183646680 x1874531145894016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145894016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145894016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697585.899212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145894144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697585.899220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.899225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183647480 x1874531145894144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145894144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145894144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697585.899241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145894272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697585.899249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697585.899253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697585/real 1787697585] req@ffff8f5183644e00 x1874531145894272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697585.899259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145894272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.899263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697585.899265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697585.899267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145894272:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697586.794675:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697587.818506:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697588.842551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697589.866527:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697590.890536:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697591.018484:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697591.018512:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697591.018517:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.018527:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:3.0:1787697591.018537:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.018544:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697591.018579:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145894400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697591.018607:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697591.018611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697591.018613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.018618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:3.0:1787697591.018621:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.018625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697591.018635:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697591.018648:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145894528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697591.018655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145894400:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697591.018664:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697591.018668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697591.018670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.018675:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:3.0:1787697591.018679:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.018683:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697591.018689:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697591.018703:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145894656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697591.018712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697591.018716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697591.018730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.018735:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:3.0:1787697591.018739:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.018742:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697591.018749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697591.018749:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697591.018764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145894784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697591.018775:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697591.018779:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697591.018781:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.018786:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:3.0:1787697591.018790:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697591.018795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f5187388380 x1874531145894400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697591.018804:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697591.018811:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697591.018813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145894400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697591.018827:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145894912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697591.018837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.018843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.018850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145894400:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697591.018862:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697591.018866:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697591.018869:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.018874:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00100000:0.0:1787697591.018879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894528:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697591.018879:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.018887:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697591.018894:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697591.018908:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145895040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697591.018912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697591.018926:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697591.018927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f518738b100 x1874531145894528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697591.018936:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697591.018941:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697591.018944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145894528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697591.018949:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:0.0:1787697591.018955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697591.018957:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697591.018961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.018967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894528:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697591.018967:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697591.018974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894656:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697591.018976:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697591.018993:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145895168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697591.019004:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697591.019005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697591.019011:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697591.019013:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.019018:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00100000:0.0:1787697591.019022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f518738b800 x1874531145894656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697591.019024:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.019034:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697591.019039:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697591.019046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145894656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697591.019054:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145895296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697591.019057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697591.019073:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697591.019074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894656:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697591.019083:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697591.019085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894784:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697591.019087:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697591.019094:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5490 00000100:00080000:3.0:1787697591.019097:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697591.019101:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697591.019106:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697591.019111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697591.019122:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145895424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697591.019127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f518738a680 x1874531145894784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697591.019150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145894784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697591.019159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.019167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697591.019181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697591.019199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697591.019210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f5187388000 x1874531145894912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697591.019226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145894912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697591.019233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.019238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145894912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697591.019254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697591.019273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697591.019284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f518738b480 x1874531145895040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697591.019294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145895040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697591.019301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.019307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697591.019312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697591.019330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697591.019341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f5187388e00 x1874531145895168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697591.019399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145895168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697591.019414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.019421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697591.019426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697591.019446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697591.019458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f5187388a80 x1874531145895296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697591.019468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145895296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697591.019475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.019480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697591.019485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697591.019514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697591.019525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697591/real 1787697591] req@ffff8f5187388380 x1874531145895424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697591.019536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145895424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697591.019544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697591.019548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697591.019551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145895424:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697591.914518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697592.938917:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697593.962591:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697594.986526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697596.010569:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697596.138422:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697596.138433:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697596.138437:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138447:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138458:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138465:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697596.138502:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145895552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697596.138528:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697596.138532:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697596.138535:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138540:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138543:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138546:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697596.138555:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697596.138569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145895680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697596.138579:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697596.138582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145895552:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697596.138583:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697596.138587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138593:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697596.138605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697596.138618:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145895808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697596.138626:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697596.138641:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697596.138643:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138647:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138651:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697596.138659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697596.138669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697596.138674:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145895936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697596.138684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697596.138687:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697596.138689:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138693:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138696:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138700:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697596.138704:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697596.138710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f5187389f80 x1874531145895552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697596.138718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145896064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697596.138729:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697596.138732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145895552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697596.138734:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697596.138738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138747:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:0.0:1787697596.138752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697596.138753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138760:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697596.138762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.138769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145895552:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697596.138769:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697596.138787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145896192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697596.138797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145895680:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697596.138800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697596.138805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697596.138807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138812:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138816:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138821:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697596.138826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697596.138827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697596.138842:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145896320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697596.138843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f518738bb80 x1874531145895680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697596.138867:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697596.138872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145895680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697596.138873:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697596.138877:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138881:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5495 00000100:00080000:3.0:1787697596.138885:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138892:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697596.138895:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697596.138898:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697596.138902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.138907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145895680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697596.138914:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145896448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697596.138916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145895808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697596.138935:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697596.138939:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697596.138942:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697596.138946:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5495 00000800:00000100:0.0:1787697596.138951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697596.138954:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697596.138961:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697596.138966:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697596.138967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f5187389c00 x1874531145895808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697596.138980:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145896576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697596.138995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145895808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145895808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697596.139016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145895936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697596.139033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697596.139042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f518738ad80 x1874531145895936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697596.139064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145895936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145895936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697596.139082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145896064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697596.139098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697596.139107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f5187389500 x1874531145896064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697596.139122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145896064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145896064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697596.139150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697596.139165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697596.139174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f518738aa00 x1874531145896192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697596.139184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145896192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697596.139202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697596.139216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697596.139225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f5187389180 x1874531145896320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697596.139234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145896320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697596.139256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697596.139270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697596.139278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f518738a300 x1874531145896448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697596.139287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145896448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697596.139304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697596.139317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697596.139326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697596/real 1787697596] req@ffff8f5187389f80 x1874531145896576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697596.139335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145896576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697596.139342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697596.139345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697596.139348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145896576:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697597.034574:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697598.058515:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697599.082549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697600.106539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697601.130665:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697601.258515:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258529:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697601.258534:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697601.258557:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258571:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258580:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697601.258631:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145896704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697601.258669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258673:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697601.258675:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697601.258681:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258685:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258689:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.258701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697601.258717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145896832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697601.258722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145896704:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697601.258733:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697601.258738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697601.258742:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258746:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258749:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.258753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697601.258764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145896960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697601.258770:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258772:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697601.258773:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697601.258775:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258777:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258780:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.258784:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697601.258790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145897088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697601.258795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697601.258801:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258804:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697601.258805:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697601.258809:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258812:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258814:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.258817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697601.258824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145897216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697601.258830:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258834:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697601.258836:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697601.258841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f5187388380 x1874531145896704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697601.258842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258848:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258856:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.258862:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697601.258871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145896704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697601.258876:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145897344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697601.258894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697601.258896:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258901:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697601.258903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697601.258905:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697601.258912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145896704:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697601.258913:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258920:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697601.258926:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.258931:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697601.258940:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145897472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697601.258946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145896832:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697601.258954:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697601.258959:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697601.258960:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697601.258966:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.258971:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697601.258973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697601.258980:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697601.258986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f5187388a80 x1874531145896832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697601.258987:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697601.259002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145896832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697601.259006:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145897600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697601.259027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697601.259029:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697601.259034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697601.259034:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697601.259039:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697601.259042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145896832:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697601.259045:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5500 00000100:00080000:3.0:1787697601.259050:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697601.259052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145896960:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697601.259057:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697601.259063:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697601.259076:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145897728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697601.259081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f5187388e00 x1874531145896960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145896960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145896960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697601.259143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145897088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697601.259159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f518738b480 x1874531145897088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145897088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145897088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697601.259199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145897216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697601.259217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f5187388000 x1874531145897216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145897216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145897216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697601.259266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697601.259287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f518738a680 x1874531145897344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145897344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697601.259344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697601.259384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f518738b800 x1874531145897472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145897472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697601.259529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697601.259561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f518738b100 x1874531145897600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145897600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697601.259596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697601.259608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697601.259615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697601/real 1787697601] req@ffff8f5187388380 x1874531145897728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697601.259622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145897728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697601.259634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697601.259638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697601.259642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145897728:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697602.154602:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697603.178536:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697604.202615:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697605.226538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697606.250500:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697606.378424:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378434:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697606.378438:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378448:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378471:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378478:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697606.378511:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145897856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697606.378542:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378546:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697606.378547:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378552:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378555:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378559:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697606.378568:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697606.378581:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145897984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697606.378591:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378594:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697606.378597:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697606.378598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145897856:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697606.378603:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378610:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378614:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697606.378618:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697606.378632:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145898112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697606.378640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697606.378645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378648:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378651:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697606.378659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697606.378666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697606.378670:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145898240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697606.378679:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378683:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697606.378684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378688:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378691:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378694:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697606.378698:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697606.378712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f5187389f80 x1874531145897856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697606.378720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145898368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697606.378729:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378733:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697606.378735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145897856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697606.378735:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378743:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378748:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378753:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697606.378756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697606.378757:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697606.378765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697606.378770:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145898496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697606.378772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145897856:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697606.378784:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378787:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697606.378788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378791:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378793:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378797:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697606.378798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145897984:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697606.378800:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697606.378810:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145898624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697606.378816:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697606.378820:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378822:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378825:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697606.378829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697606.378831:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697606.378835:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697606.378843:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145898752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697606.378849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f518738a300 x1874531145897984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697606.378850:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697606.378856:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697606.378859:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697606.378864:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5506 00000100:00080000:3.0:1787697606.378870:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697606.378873:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697606.378876:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697606.378877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145897984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697606.378883:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145898880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697606.378893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.378900:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.378904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145897984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697606.378911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145898112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697606.378929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.378940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f5187389180 x1874531145898112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.378950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145898112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.378957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.378960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.378963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145898112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697606.378968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145898240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697606.378986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.378996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f518738aa00 x1874531145898240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.379005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145898240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.379012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.379015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.379018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145898240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697606.379024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145898368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697606.379041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.379051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f5187389500 x1874531145898368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.379067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145898368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.379074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.379077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.379080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145898368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697606.379086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697606.379103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.379113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f518738ad80 x1874531145898496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.379124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145898496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.379132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.379136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.379139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697606.379155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697606.379172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.379182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f5187389c00 x1874531145898624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.379193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145898624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.379206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.379209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.379212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697606.379217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697606.379233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.379243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f518738bb80 x1874531145898752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.379253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145898752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.379259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.379262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.379264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697606.379270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697606.379287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697606.379297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697606/real 1787697606] req@ffff8f5187389f80 x1874531145898880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697606.379307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145898880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697606.379315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697606.379317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697606.379321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145898880:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697607.274554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697608.298535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697609.322623:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697610.346516:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697611.370571:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697611.498504:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498512:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697611.498515:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498525:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498535:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498542:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697611.498575:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145899008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498612:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697611.498614:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498622:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498635:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697611.498646:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145899136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498655:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498658:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697611.498660:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498663:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498666:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498669:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697611.498684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145899264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498695:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697611.498697:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498703:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498706:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498711:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697611.498719:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145899392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697611.498732:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498735:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498738:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498741:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697611.498756:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145899520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498764:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697611.498768:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498772:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498775:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498778:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498783:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697611.498792:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145899648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498803:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697611.498805:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498809:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498815:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498819:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697611.498828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145899776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498839:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697611.498841:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498844:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498846:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498850:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498854:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697611.498862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145899904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697611.498870:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697611.498872:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697611.498874:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697611.498877:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5511 00000100:00080000:1.0:1787697611.498880:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697611.498883:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697611.498886:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697611.498895:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145900032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697611.499048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145899008:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787697611.499107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183647800 x1874531145899008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145899008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145899008:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697611.499181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697611.499199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183647100 x1874531145899136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145899136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697611.499242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697611.499261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183645180 x1874531145899264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145899264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697611.499301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697611.499318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183645c00 x1874531145899392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145899392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697611.499395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697611.499416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183644e00 x1874531145899520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145899520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145899520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697611.499470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145899648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697611.499488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183647480 x1874531145899648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145899648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145899648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697611.499528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145899776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697611.499545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183646680 x1874531145899776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145899776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145899776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697611.499589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145899904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697611.499607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183644000 x1874531145899904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145899904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145899904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697611.499645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145900032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697611.499660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697611.499669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697611/real 1787697611] req@ffff8f5183644700 x1874531145900032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697611.499679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145900032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697611.499686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697611.499689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697611.499692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145900032:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697612.394499:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697613.418489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697614.442520:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697615.466535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697616.490488:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697616.618467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618474:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697616.618477:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618485:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618498:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697616.618527:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145900160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618545:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618548:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697616.618549:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618552:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618554:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618557:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618564:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697616.618571:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145900288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618577:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697616.618586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697616.618597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145900160:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697616.618597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618603:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697616.618612:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145900416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618618:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618620:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697616.618621:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618623:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618625:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618626:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697616.618636:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145900544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697616.618645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618648:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618650:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618652:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697616.618660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145900672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618665:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697616.618668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618671:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618673:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618675:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618677:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697616.618684:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145900800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618689:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697616.618692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618698:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00100000:0.0:1787697616.618699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900288:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697616.618700:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618706:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618710:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697616.618719:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145900928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697616.618726:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618730:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697616.618731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618735:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618739:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697616.618744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900416:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697616.618745:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618753:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697616.618768:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145901056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697616.618776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900544:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697616.618779:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697616.618783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697616.618784:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697616.618788:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5516 00000100:00080000:3.0:1787697616.618791:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697616.618794:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697616.618798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697616.618805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697616.618809:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145901184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697616.618830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145900800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697616.618848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145900928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697616.618866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145901056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697616.618881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145901184:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697617.514528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697618.538524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697619.562532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697619.690612:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787697619.690649:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697619.690663:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690708:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690720:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690729:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690739:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690750:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787697619.690751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f5187389f80 x1874531145901184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697619.690758:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690767:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697619.690777:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787697619.690792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145901184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.690805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.690811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.690816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145901184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697619.690824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f518738b800 x1874531145900416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.690838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145900416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.690846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.690850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.690853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697619.690860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f518738a680 x1874531145900544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.690871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145900544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.690880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.690897:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.690900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697619.690907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f5187388000 x1874531145900672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.690919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145900672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.690927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.690930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.690934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697619.690942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f518738b480 x1874531145900800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.690953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145900800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.690962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.690965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.690968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145900800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697619.690975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f5187388e00 x1874531145900928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.690986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145900928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.690993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.690997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.691000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145900928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697619.691006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f5187388380 x1874531145900160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.691018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145900160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.691032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.691036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.691039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145900160:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697619.691047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f518738b100 x1874531145900288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.691058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145900288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.691066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.691069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.691072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145900288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697619.691104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697616/real 1787697619] req@ffff8f5187388a80 x1874531145901056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697619.691117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145901056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697619.691124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697619.691127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697619.691130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145901056:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697620.586540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697621.610546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697621.738475:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738485:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697621.738488:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.738498:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738508:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738516:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697621.738552:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145901312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697621.738578:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738581:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697621.738582:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.738587:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738591:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738595:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697621.738604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697621.738617:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145901440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697621.738627:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738630:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697621.738632:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.738636:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738639:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738645:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697621.738650:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697621.738666:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145901568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697621.738676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145901312:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697621.738688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697621.738695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.738700:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738704:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738709:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697621.738715:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697621.738732:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145901696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697621.738741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738745:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697621.738746:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.738750:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738754:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738758:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697621.738763:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697621.738766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697621.738780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145901824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697621.738791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738794:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697621.738796:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.738801:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738804:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738808:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697621.738815:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697621.738828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f518738bb80 x1874531145901312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697621.738830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145901952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697621.738896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.738899:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697621.738901:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697621.738907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145901312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697621.738914:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.738926:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.738940:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697621.738941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697621.738949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697621.738953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.738964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145901312:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697621.738975:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145902080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697621.738980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901440:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697621.739003:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697621.739007:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697621.739009:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.739015:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.739020:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.739025:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697621.739036:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697621.739048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697621.739056:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145902208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697621.739071:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697621.739080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f5187389c00 x1874531145901440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697621.739081:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697621.739086:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697621.739102:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5521 00000100:00080000:3.0:1787697621.739117:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697621.739129:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145901440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697621.739142:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697621.739154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697621.739167:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145902336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697621.739174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697621.739244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697621.739275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f518738ad80 x1874531145901568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145901568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697621.739322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697621.739345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f5187389500 x1874531145901696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145901696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697621.739433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697621.739453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f518738aa00 x1874531145901824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145901824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145901824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697621.739496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145901952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697621.739516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f5187389180 x1874531145901952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145901952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145901952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697621.739564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145902080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697621.739584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f518738a300 x1874531145902080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145902080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145902080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697621.739625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145902208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697621.739644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f5187388a80 x1874531145902208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145902208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145902208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697621.739695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145902336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697621.739715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697621.739724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697621/real 1787697621] req@ffff8f518738b100 x1874531145902336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697621.739732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145902336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697621.739738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697621.739741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697621.739744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145902336:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697622.634536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697623.658539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697624.682517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697625.706514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697626.730511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697626.858445:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858452:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697626.858455:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858462:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858470:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858476:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697626.858504:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145902464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697626.858534:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858538:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697626.858541:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858545:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858548:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858552:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697626.858561:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697626.858576:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145902592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697626.858582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145902464:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697626.858592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697626.858597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858608:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858613:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858617:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697626.858622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697626.858643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697626.858645:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145902720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697626.858657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858660:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697626.858666:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697626.858675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f5187388380 x1874531145902464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.858679:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858687:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858693:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697626.858695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145902464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697626.858701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697626.858703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.858708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.858710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145902464:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697626.858717:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145902848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697626.858728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697626.858729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697626.858736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858739:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858743:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697626.858747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697626.858749:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697626.858754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697626.858759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f5187388e00 x1874531145902592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697626.858765:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145902976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697626.858775:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697626.858778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145902592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.858778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697626.858781:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858786:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858790:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858795:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697626.858799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697626.858799:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697626.858807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697626.858810:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145903104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697626.858815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697626.858819:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858822:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697626.858826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697626.858826:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858834:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858836:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858838:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697626.858842:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697626.858849:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145903232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697626.858853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697626.858861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697626.858866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f518738b480 x1874531145902720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.858867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858875:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858880:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697626.858884:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697626.858886:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145902720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697626.858888:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697626.858898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697626.858902:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145903360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697626.858904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.858910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697626.858918:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.858921:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697626.858922:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.858925:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5526 00000100:00080000:3.0:1787697626.858927:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697626.858928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902848:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697626.858932:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697626.858936:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697626.858945:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145903488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697626.858951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697626.858964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f5187388000 x1874531145902848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697626.858972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145902848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697626.858978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.858981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.858983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697626.858987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697626.858995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697626.859001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f518738a680 x1874531145902976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697626.859007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145902976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697626.859011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.859013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.859014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145902976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697626.859017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697626.859026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697626.859031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f518738b800 x1874531145903104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697626.859041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145903104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697626.859046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.859047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.859049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697626.859052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697626.859060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697626.859064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f5187389f80 x1874531145903232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697626.859070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145903232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697626.859075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.859076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.859078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697626.859080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697626.859088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697626.859093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f5187388380 x1874531145903360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697626.859098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145903360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697626.859128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.859130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.859132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697626.859143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697626.859155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697626.859161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697626/real 1787697626] req@ffff8f518738b100 x1874531145903488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697626.859171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145903488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697626.859178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697626.859180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697626.859183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145903488:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697627.754486:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697628.778535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697629.802465:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697630.826561:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697631.850523:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697631.978460:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978467:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697631.978470:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978483:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697631.978498:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697631.978529:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145903616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697631.978549:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978552:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697631.978554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978566:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697631.978569:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978579:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697631.978593:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145903744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697631.978601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145903616:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697631.978616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697631.978620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978627:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697631.978629:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978633:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697631.978644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145903872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697631.978650:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697631.978653:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697631.978659:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697631.978666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697631.978674:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145904000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697631.978681:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978683:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697631.978684:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978686:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978688:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697631.978690:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697631.978702:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145904128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697631.978706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f5187388a80 x1874531145903616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697631.978712:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978718:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697631.978722:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978725:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978727:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697631.978730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145903616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697631.978731:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978737:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697631.978747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697631.978749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145904256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697631.978753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.978760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145903616:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697631.978763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697631.978768:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978771:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978773:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697631.978775:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697631.978783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145903744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697631.978785:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145904384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697631.978794:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978796:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697631.978797:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697631.978800:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:3.0:1787697631.978803:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697631.978806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697631.978809:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697631.978814:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697631.978820:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145904512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697631.978829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f518738a300 x1874531145903744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697631.978830:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697631.978839:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697631.978844:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697631.978847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145903744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697631.978851:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5531 00000100:00080000:0.0:1787697631.978857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697631.978859:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697631.978861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.978865:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145903744:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697631.978868:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697631.978872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145903872:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697631.978874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697631.978888:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145904640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697631.978893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.978906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f5187389180 x1874531145903872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.978915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145903872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.978921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.978923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.978926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145903872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697631.978930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145904000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697631.978940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.978946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f518738aa00 x1874531145904000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.978952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145904000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.978956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.978958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.978959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145904000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697631.978962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145904128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697631.978974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.978983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f5187389500 x1874531145904128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.978994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145904128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.979001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.979005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.979008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145904128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697631.979025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697631.979041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.979050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f518738ad80 x1874531145904256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.979061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145904256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.979069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.979072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.979075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697631.979091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697631.979107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.979115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f5187389c00 x1874531145904384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.979125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145904384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.979132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.979135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.979138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697631.979143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697631.979160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.979169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f518738bb80 x1874531145904512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.979182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145904512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.979190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.979193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.979196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697631.979202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697631.979218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697631.979228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697631/real 1787697631] req@ffff8f5187388a80 x1874531145904640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697631.979239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145904640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697631.979246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697631.979249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697631.979252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145904640:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697632.874506:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697633.898544:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697634.922586:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697635.946605:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697636.970550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697637.098434:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098443:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697637.098448:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098457:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098467:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098474:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697637.098507:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145904768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697637.098534:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098537:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697637.098546:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098553:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098557:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098561:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697637.098570:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697637.098582:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145904896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697637.098599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145904768:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697637.098603:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098609:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697637.098611:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098615:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098620:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098624:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697637.098628:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697637.098643:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145905024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697637.098652:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697637.098657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098662:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098666:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697637.098670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697637.098672:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697637.098678:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697637.098691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145905152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697637.098700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697637.098705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697637.098708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f518738b100 x1874531145904768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697637.098712:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098718:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098724:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697637.098729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145904768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697637.098729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697637.098741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697637.098744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145905280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697637.098747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.098753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145904768:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697637.098764:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697637.098769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098773:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098777:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098780:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697637.098785:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697637.098792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145904896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697637.098797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145905408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697637.098811:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098814:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697637.098816:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098820:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098824:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697637.098828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697637.098832:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697637.098843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f5187388380 x1874531145904896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697637.098845:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697637.098853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145904896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.098860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.098865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697637.098865:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145905536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697637.098871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145904896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697637.098882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145905024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697637.098886:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098890:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697637.098892:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098896:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00080000:3.0:1787697637.098901:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098905:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697637.098910:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697637.098913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697637.098935:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145905664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697637.098944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f5187389f80 x1874531145905024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697637.098946:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697637.098953:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697637.098957:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697637.098965:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5536 00000100:00100000:0.0:1787697637.098970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145905024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697637.098971:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697637.098978:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697637.098983:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697637.098991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.098995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697637.099000:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145905792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697637.099001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145905024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697637.099013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145905152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697637.099038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697637.099059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f518738b800 x1874531145905152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697637.099073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145905152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.099081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.099083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.099086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145905152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697637.099091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145905280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697637.099111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697637.099122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f518738a680 x1874531145905280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697637.099138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145905280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.099146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.099149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.099152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145905280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697637.099169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697637.099196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697637.099206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f5187388000 x1874531145905408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697637.099217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145905408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.099229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.099231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.099233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697637.099237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697637.099263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697637.099273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f518738b480 x1874531145905536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697637.099283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145905536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.099291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.099294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.099297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697637.099302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697637.099319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697637.099327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f5187388e00 x1874531145905664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697637.099337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145905664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.099344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.099347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.099350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697637.099380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697637.099401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697637.099411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697637/real 1787697637] req@ffff8f518738b100 x1874531145905792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697637.099421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145905792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697637.099429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697637.099432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697637.099434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145905792:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697637.994558:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697639.018569:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697640.042496:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697641.066485:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697642.090483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697642.218465:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697642.218480:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697642.218487:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218496:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218501:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697642.218533:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145905920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697642.218560:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218563:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697642.218565:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697642.218569:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218573:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218577:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697642.218586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697642.218598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145905920:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697642.218600:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145906048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697642.218613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697642.218618:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697642.218623:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218626:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218630:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697642.218635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697642.218647:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145906176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697642.218651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697642.218661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697642.218667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697642.218672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218679:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697642.218684:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697642.218686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f5187388a80 x1874531145905920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697642.218697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145906304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697642.218702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145905920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697642.218719:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697642.218721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697642.218727:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697642.218728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697642.218732:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697642.218734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145905920:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697642.218741:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218746:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218750:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697642.218754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906048:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697642.218755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697642.218772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145906432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697642.218780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697642.218791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218794:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697642.218795:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697642.218799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f518738bb80 x1874531145906048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697642.218800:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218805:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218810:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697642.218816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697642.218824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145906048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697642.218827:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145906560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697642.218851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218853:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697642.218854:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697642.218857:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.218861:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218868:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697642.218869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.218877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697642.218877:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697642.218884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697642.218894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697642.218894:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145906688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697642.218914:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697642.218917:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697642.218919:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697642.218923:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5541 00000800:00000100:0.0:1787697642.218928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697642.218930:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697642.218935:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697642.218940:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697642.218941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f5187389c00 x1874531145906176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697642.218954:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145906816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697642.218955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145906176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697642.218981:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697642.218982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697642.218987:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697642.218990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697642.218991:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697642.218995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906176:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697642.218996:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5541 00000100:00080000:3.0:1787697642.219002:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697642.219006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906304:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697642.219006:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697642.219013:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697642.219022:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145906944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697642.219028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697642.219037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f518738ad80 x1874531145906304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697642.219048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145906304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697642.219052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.219054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697642.219056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697642.219065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697642.219074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697642.219080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f5187389500 x1874531145906432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697642.219092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145906432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697642.219097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.219099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697642.219101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145906432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697642.219104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697642.219113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697642.219119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f518738aa00 x1874531145906560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697642.219125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145906560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697642.219133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.219135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697642.219137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697642.219140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697642.219147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697642.219152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f5187389180 x1874531145906688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697642.219158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145906688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697642.219162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.219164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697642.219165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697642.219168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697642.219176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697642.219180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f518738a300 x1874531145906816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697642.219186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145906816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697642.219190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.219192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697642.219194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697642.219197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697642.219205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697642.219209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697642/real 1787697642] req@ffff8f5187388a80 x1874531145906944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697642.219215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145906944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697642.219219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697642.219220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697642.219222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145906944:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697643.114504:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697644.138504:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697645.162537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697646.186532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697647.210650:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697647.338477:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338487:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697647.338492:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338501:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338511:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697647.338518:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697647.338551:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145907072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697647.338573:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338575:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697647.338576:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338579:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338582:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697647.338585:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338591:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697647.338599:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145907200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697647.338606:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338608:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697647.338609:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338611:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338613:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697647.338618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145907072:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697647.338618:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338624:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697647.338632:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145907328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697647.338642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697647.338645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338648:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338650:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697647.338652:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338655:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697647.338662:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145907456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697647.338667:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697647.338671:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338673:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697647.338677:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697647.338682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338683:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697647.338692:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145907584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697647.338698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697647.338702:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338704:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338706:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697647.338708:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697647.338718:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145907712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697647.338723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338725:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697647.338726:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338728:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338730:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697647.338734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f518738b100 x1874531145907072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697647.338734:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697647.338763:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145907840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697647.338770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145907072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697647.338771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697647.338780:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338788:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338793:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697647.338794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697647.338797:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697647.338801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697647.338802:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697647.338809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145907072:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697647.338813:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145907968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697647.338820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907200:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697647.338831:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697647.338834:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697647.338835:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697647.338838:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5546 00000100:00080000:3.0:1787697647.338840:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697647.338843:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697647.338846:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697647.338851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697647.338855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145908096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697647.338868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f5187388e00 x1874531145907200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.338883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145907200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.338891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.338894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.338897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697647.338918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697647.338936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.338945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f518738b480 x1874531145907328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.338955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145907328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.338962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.338965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.338968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697647.338973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697647.338987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.338996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f5187388000 x1874531145907456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.339011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145907456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.339018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.339021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.339023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697647.339028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697647.339042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.339050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f518738a680 x1874531145907584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.339060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145907584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.339067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.339070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.339073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145907584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697647.339078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145907712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697647.339092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.339100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f518738b800 x1874531145907712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.339110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145907712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.339123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.339125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.339128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145907712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697647.339133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145907840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697647.339147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.339155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f5187389f80 x1874531145907840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.339165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145907840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.339172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.339174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.339177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145907840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697647.339183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145907968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697647.339196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.339205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f5187388380 x1874531145907968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.339215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145907968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.339222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.339224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.339227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145907968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697647.339232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145908096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697647.339246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697647.339255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697647/real 1787697647] req@ffff8f5187388a80 x1874531145908096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697647.339266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145908096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697647.339272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697647.339275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697647.339278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145908096:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697648.234523:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697649.258494:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697650.282525:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697651.306493:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697652.330511:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697652.458409:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458417:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697652.458420:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697652.458427:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:3.0:1787697652.458435:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458440:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697652.458466:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145908224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697652.458487:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458490:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697652.458492:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697652.458494:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:3.0:1787697652.458497:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458500:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458509:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697652.458527:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145908352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697652.458530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145908224:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697652.458544:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458547:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697652.458549:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697652.458554:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:3.0:1787697652.458558:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458561:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458566:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697652.458578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145908480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697652.458587:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697652.458592:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697652.458595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697652.458599:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:3.0:1787697652.458604:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458609:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458614:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697652.458626:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145908608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697652.458633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f518738a300 x1874531145908224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697652.458641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458646:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697652.458648:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697652.458650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145908224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697652.458655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:0.0:1787697652.458673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697652.458674:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697652.458679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697652.458682:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697652.458685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145908224:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697652.458690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697652.458704:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145908736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697652.458705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697652.458723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458727:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697652.458729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697652.458734:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5552 00000800:00000100:0.0:1787697652.458739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697652.458740:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458747:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697652.458756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f5187389180 x1874531145908352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697652.458766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145908864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697652.458771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145908352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697652.458786:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697652.458787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.458792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697652.458795:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697652.458796:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697652.458799:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697652.458803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697652.458807:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:3.0:1787697652.458812:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458815:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697652.458820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458820:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697652.458831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f518738aa00 x1874531145908480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697652.458833:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145908992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697652.458852:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697652.458857:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697652.458858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145908480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697652.458861:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00080000:3.0:1787697652.458865:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458869:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697652.458876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697652.458881:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145909120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697652.458883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.458889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697652.458892:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697652.458897:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697652.458899:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697652.458902:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5552 00000100:00100000:0.0:1787697652.458903:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908608:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697652.458904:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697652.458908:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697652.458913:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697652.458920:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145909248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697652.458934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697652.458948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f5187389500 x1874531145908608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697652.458965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145908608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697652.458970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.458971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.458974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697652.458984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697652.458996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697652.459002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f518738ad80 x1874531145908736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697652.459008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145908736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697652.459013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.459015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.459017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145908736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697652.459020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145908864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697652.459028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697652.459033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f5187389c00 x1874531145908864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697652.459039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145908864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697652.459048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.459049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.459051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145908864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697652.459054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145908992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697652.459062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697652.459067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f518738bb80 x1874531145908992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697652.459073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145908992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697652.459077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.459079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.459080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145908992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697652.459083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145909120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697652.459091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697652.459095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f518738a300 x1874531145909120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697652.459101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145909120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697652.459105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.459106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.459108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145909120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697652.459111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145909248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697652.459119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697652.459124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697652/real 1787697652] req@ffff8f5187388a80 x1874531145909248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697652.459129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145909248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697652.459133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697652.459134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697652.459136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145909248:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697653.354463:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697654.378523:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697655.402564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697656.426488:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697657.450582:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697657.578511:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578521:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697657.578525:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578534:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578546:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578552:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697657.578616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145909376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578639:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578642:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697657.578643:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578647:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578650:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578653:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697657.578672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145909504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697657.578683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578689:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00100000:0.0:1787697657.578690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145909376:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697657.578691:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578697:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578702:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697657.578707:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145909632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697657.578717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578719:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578721:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578723:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578727:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697657.578733:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145909760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578743:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578745:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697657.578747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578749:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578751:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578753:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578756:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697657.578760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697657.578765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145909888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578772:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697657.578777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578782:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578784:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697657.578793:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145910016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578799:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578801:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697657.578803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697657.578807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183646d80 x1874531145909376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697657.578808:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578813:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578816:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578819:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697657.578829:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145910144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697657.578837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145909376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697657.578842:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697657.578845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578852:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:0.0:1787697657.578854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697657.578855:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697657.578860:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697657.578862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697657.578864:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697657.578867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145909376:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697657.578875:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145910272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697657.578881:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697657.578883:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697657.578884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697657.578887:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5557 00000100:00080000:1.0:1787697657.578890:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697657.578894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909504:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697657.578895:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697657.578901:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697657.578913:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145910400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697657.578931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.578942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183646a00 x1874531145909504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.578950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145909504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.578956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.578958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.578960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697657.578964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697657.578974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.578985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183647b80 x1874531145909632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.578992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145909632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.578997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.578999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697657.579004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697657.579015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.579020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183644380 x1874531145909760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.579036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145909760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.579041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.579042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697657.579047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697657.579057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.579062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183644700 x1874531145909888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.579068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145909888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.579078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.579080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145909888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697657.579085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697657.579095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.579101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183644000 x1874531145910016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.579107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145910016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.579117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.579120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697657.579128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697657.579146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.579156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183646680 x1874531145910144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.579168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145910144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.579174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.579176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697657.579182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697657.579196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.579202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183647480 x1874531145910272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.579209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145910272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.579213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.579214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697657.579237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697657.579249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697657.579255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697657/real 1787697657] req@ffff8f5183644e00 x1874531145910400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697657.579261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145910400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697657.579266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697657.579268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697657.579269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145910400:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697658.474594:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697659.498603:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697660.522506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697661.546548:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697662.570568:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697662.698476:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697662.698503:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698517:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698530:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698538:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697662.698585:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145910528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697662.698640:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698645:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698650:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698654:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697662.698680:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145910656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697662.698695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698698:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698701:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698705:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697662.698719:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145910784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697662.698729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698731:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698734:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698736:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697662.698748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145910912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698755:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697662.698759:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698767:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698770:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698775:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697662.698787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145911040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698798:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697662.698800:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698805:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698808:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698812:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698817:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697662.698828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145911168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698836:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698839:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697662.698841:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698844:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698860:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698864:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698867:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697662.698875:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145911296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698881:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698883:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697662.698884:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698887:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698889:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698891:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697662.698899:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145911424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697662.698908:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697662.698910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697662.698912:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697662.698923:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5562 00000100:00080000:1.0:1787697662.698929:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697662.698933:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697662.698938:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697662.698949:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145911552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697662.699066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145910528:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787697662.699148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183645c00 x1874531145910528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145910528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145910528:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697662.699242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145910656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697662.699267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183645180 x1874531145910656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145910656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145910656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697662.699310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145910784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697662.699328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183647100 x1874531145910784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145910784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145910784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697662.699402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145910912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697662.699418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183647800 x1874531145910912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145910912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145910912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697662.699458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145911040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697662.699477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183644e00 x1874531145911040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145911040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145911040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697662.699530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697662.699545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183647480 x1874531145911168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145911168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697662.699587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697662.699605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183646680 x1874531145911296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145911296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697662.699645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697662.699659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183644000 x1874531145911424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145911424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697662.699686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697662.699695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697662.699700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697662/real 1787697662] req@ffff8f5183644700 x1874531145911552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697662.699706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145911552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697662.699711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697662.699713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697662.699714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145911552:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697663.594506:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697664.618544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697665.642537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697666.666555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697667.690528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697667.818530:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.818543:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697667.818548:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697667.818562:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.818576:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697667.818585:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697667.818635:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145911680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697667.818668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.818672:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697667.818674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697667.818678:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.818682:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697667.818686:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697667.818698:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697667.818710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145911808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697667.818720:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.818724:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697667.818725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697667.818729:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.818732:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697667.818736:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697667.818741:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697667.818755:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145911936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697667.818756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145911680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697667.818779:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.818783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697667.818786:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697667.818792:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.818796:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697667.818801:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697667.818807:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697667.818821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145912064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697667.818832:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.818839:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697667.818841:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697667.818846:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5567 00000800:00000100:0.0:1787697667.818847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697667.818851:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697667.818859:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697667.818865:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697667.818881:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145912192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697667.818892:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.818896:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697667.818898:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697667.818905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183644380 x1874531145911680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697667.818905:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.818914:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697667.818925:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697667.818932:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697667.818933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145911680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697667.818949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.818953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.818957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145911680:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697667.818964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145911808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697667.818968:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145912320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697667.818986:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697667.818988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697667.818995:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697667.818998:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697667.819003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183647b80 x1874531145911808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697667.819010:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00100000:0.0:1787697667.819020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145911808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697667.819021:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697667.819034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697667.819035:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697667.819043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145911808:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697667.819048:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697667.819069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145911936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697667.819069:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145912448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697667.819093:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697667.819096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697667.819102:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697667.819105:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697667.819109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183646a00 x1874531145911936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697667.819116:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.819127:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697667.819129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145911936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697667.819140:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697667.819141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145911936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697667.819150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145912064:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697667.819163:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697667.819164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697667.819172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183646d80 x1874531145912064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697667.819186:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145912576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697667.819190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145912064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697667.819238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145912064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697667.819246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145912192:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697667.819258:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697667.819264:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697667.819269:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697667.819271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697667.819279:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5567 00000100:00080000:1.0:1787697667.819285:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697667.819290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183644700 x1874531145912192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697667.819295:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697667.819308:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697667.819320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145912192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697667.819327:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145912704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697667.819336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145912192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697667.819409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697667.819432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697667.819445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183644000 x1874531145912320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697667.819456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145912320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697667.819470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697667.819493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697667.819512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697667.819522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183646680 x1874531145912448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697667.819529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145912448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697667.819533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697667.819541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697667.819552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697667.819557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183647480 x1874531145912576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697667.819564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145912576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697667.819568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697667.819577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697667.819592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697667.819602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697667/real 1787697667] req@ffff8f5183644e00 x1874531145912704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697667.819613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145912704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697667.819621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697667.819624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697667.819626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145912704:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697668.714555:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697669.738563:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697670.762478:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697671.786543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697672.810528:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697672.938471:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938487:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697672.938493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938503:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938514:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938521:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697672.938560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145912832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938589:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697672.938590:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938594:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938598:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938606:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697672.938614:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145912960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938621:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938624:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697672.938626:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938628:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938630:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697672.938642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145912832:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697672.938643:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145913088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938666:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697672.938668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938670:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938672:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938674:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938678:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697672.938685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145913216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697672.938694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938702:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938704:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697672.938710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697672.938714:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145913344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938721:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938723:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697672.938725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938728:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938731:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938733:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697672.938742:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145913472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697672.938750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938753:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938755:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938757:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938760:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697672.938766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145913600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697672.938771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697672.938773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183647800 x1874531145912832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697672.938774:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697672.938777:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697672.938783:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938787:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697672.938792:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938795:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697672.938802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145913728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697672.938806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145912832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697672.938825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697672.938829:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697672.938831:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697672.938832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697672.938835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5572 00000100:00080000:1.0:1787697672.938839:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697672.938841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697672.938844:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697672.938848:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697672.938851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145912832:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697672.938857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145913856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697672.938863:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145912960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697672.938892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.938905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183647100 x1874531145912960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.938917:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145912960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.938924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.938927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.938930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145912960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697672.938955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145913088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697672.938969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.938976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183645180 x1874531145913088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.938983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145913088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.938988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.938990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.938992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145913088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697672.938995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145913216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697672.939005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.939011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183645c00 x1874531145913216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.939022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145913216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.939026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.939028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.939029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145913216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697672.939032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145913344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697672.939041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.939046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183644e00 x1874531145913344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.939052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145913344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.939057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.939058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.939060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145913344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697672.939063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697672.939072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.939077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183647480 x1874531145913472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.939083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145913472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.939093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.939094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.939096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697672.939099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697672.939107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.939112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183646680 x1874531145913600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.939118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145913600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.939123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.939124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.939126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697672.939129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697672.939145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.939154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183644000 x1874531145913728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.939174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145913728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.939180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.939183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.939186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697672.939192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697672.939210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697672.939220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff8f5183644700 x1874531145913856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697672.939227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145913856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697672.939232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697672.939234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697672.939236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145913856:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697673.834527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697674.858488:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697675.882513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697676.906482:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697677.930567:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697678.058481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697678.058494:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697678.058499:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058512:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058526:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.058535:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697678.058583:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145913984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697678.058616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697678.058621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697678.058623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058628:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058632:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.058637:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697678.058648:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697678.058662:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145914112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697678.058673:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697678.058677:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697678.058679:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058683:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058687:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.058692:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697678.058693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145913984:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697678.058698:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697678.058714:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145914240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697678.058725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697678.058729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697678.058731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058735:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058740:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.058744:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697678.058750:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697678.058764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697678.058764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145914368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697678.058788:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697678.058792:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697678.058794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058799:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058803:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697678.058809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f5187388380 x1874531145913984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697678.058810:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697678.058819:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697678.058829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145913984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697678.058835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145914496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697678.058840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.058848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.058853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145913984:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697678.058853:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697678.058862:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697678.058865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697678.058867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058873:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058876:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.058883:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697678.058890:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697678.058900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697678.058906:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145914624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697678.058911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f5187389f80 x1874531145914112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697678.058930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145914112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697678.058941:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697678.058942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.058947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697678.058949:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697678.058952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697678.058954:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.058961:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.058966:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.058971:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697678.058976:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697678.058978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697678.058993:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145914752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697678.058997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f518738b800 x1874531145914240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697678.059010:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697678.059016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145914240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697678.059016:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697678.059020:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.059028:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.059033:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697678.059034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.059039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697678.059042:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697678.059052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914368:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787697678.059052:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697678.059071:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145914880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697678.059074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697678.059085:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697678.059089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f518738a680 x1874531145914368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697678.059090:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697678.059094:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697678.059101:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5577 00000100:00080000:3.0:1787697678.059107:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697678.059114:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697678.059121:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697678.059123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145914368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697678.059134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697678.059137:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145915008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697678.059140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.059149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697678.059159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697678.059181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f5187388000 x1874531145914496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697678.059203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145914496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697678.059211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.059213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.059246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145914496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697678.059254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145914624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697678.059280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f518738b480 x1874531145914624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697678.059295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145914624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697678.059305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.059307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.059309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145914624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697678.059320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145914752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697678.059332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f5187388e00 x1874531145914752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697678.059344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145914752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697678.059348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.059350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.059351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145914752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697678.059391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145914880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697678.059402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f518738b100 x1874531145914880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697678.059416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145914880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697678.059420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.059422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.059424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145914880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697678.059428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145915008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697678.059436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697678.059441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697678/real 1787697678] req@ffff8f5187389f80 x1874531145915008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697678.059447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145915008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697678.059452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697678.059453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697678.059454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145915008:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697678.954526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697679.978520:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697681.002516:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697682.026548:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697683.050493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697683.178491:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178505:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697683.178510:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178522:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178537:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178545:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697683.178589:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145915136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697683.178620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178623:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178626:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178630:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.178641:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697683.178655:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145915264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178665:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178669:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697683.178672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178676:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178681:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178687:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.178693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697683.178699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145915136:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697683.178712:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145915392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178726:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697683.178727:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178730:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178733:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178736:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.178742:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697683.178751:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145915520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178757:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697683.178760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178765:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178767:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.178770:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697683.178778:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145915648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178784:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178788:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697683.178789:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178794:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178797:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178802:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697683.178816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915264:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697683.178818:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697683.178849:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145915776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178863:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697683.178864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915392:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697683.178875:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697683.178878:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697683.178886:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178894:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697683.178898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915520:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697683.178904:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.178912:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697683.178921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697683.178928:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145915904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.178936:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.178939:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697683.178941:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697683.178947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145915776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697683.178951:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.178966:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.178974:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.178979:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697683.178981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145915904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697683.179000:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145916032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697683.179012:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697683.179015:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697683.179017:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697683.179018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145916032:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697683.179024:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5582 00000100:00080000:3.0:1787697683.179029:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697683.179033:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697683.179037:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697683.179045:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145916160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697683.179062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145916160:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697684.074517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697685.098588:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697686.122527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697686.250636:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:0.0:1787697686.250685:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697686.250705:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787697686.250760:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:0.0:1787697686.250776:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787697686.250787:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787697686.250793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f5187388380 x1874531145915136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697686.250803:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787697686.250816:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787697686.250832:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787697686.250832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145915136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697686.250843:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787697686.250854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1787697686.250855:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787697686.250864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.250873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145915136:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697686.250884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f5187388a80 x1874531145915264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.250897:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145915264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.250905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.250910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.250913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697686.250945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f5187389180 x1874531145916160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.250958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145916160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.250967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.250971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.250974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145916160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697686.250981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f518738aa00 x1874531145916032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.250993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145916032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.251001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.251005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.251008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145916032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697686.251014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f518738ad80 x1874531145915776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.251027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145915776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.251036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.251041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.251044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145915776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697686.251052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f5187389500 x1874531145915904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.251063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145915904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.251071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.251084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.251088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145915904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697686.251096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f518738a300 x1874531145915392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.251110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145915392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.251118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.251122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.251125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697686.251131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f518738bb80 x1874531145915520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.251146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145915520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.251152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.251157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.251159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697686.251166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697683/real 1787697686] req@ffff8f5187389c00 x1874531145915648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697686.251177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145915648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697686.251184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697686.251188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697686.251191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145915648:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697687.146555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697688.170571:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697688.298467:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298477:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697688.298481:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298491:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298501:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697688.298508:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697688.298546:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145916288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697688.298572:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298576:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697688.298578:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298585:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697688.298589:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697688.298598:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697688.298610:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145916416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697688.298619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145916288:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697688.298627:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298632:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697688.298634:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298638:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298642:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697688.298646:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697688.298650:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697688.298662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145916544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697688.298671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697688.298677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298680:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298684:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697688.298687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697688.298691:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697688.298696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697688.298706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145916672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697688.298716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697688.298721:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298726:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298729:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697688.298733:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697688.298738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f5187389c00 x1874531145916288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697688.298738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697688.298751:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145916800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697688.298761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697688.298763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145916288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697688.298766:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697688.298770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:1.0:1787697688.298782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697688.298783:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697688.298790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697688.298791:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697688.298798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145916288:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697688.298799:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697688.298815:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145916928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697688.298830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916416:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697688.298834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298839:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697688.298846:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298851:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298855:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697688.298860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697688.298863:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697688.298870:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697688.298879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f518738bb80 x1874531145916416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697688.298884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145917056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697688.298905:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697688.298910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145916416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697688.298911:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697688.298914:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697688.298923:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298928:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697688.298929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697688.298935:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697688.298936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.298942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916416:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697688.298944:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697688.298952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697688.298958:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145917184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697688.298970:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697688.298974:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697688.298976:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697688.298981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697688.298983:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5587 00000100:00080000:3.0:1787697688.298989:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697688.298994:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697688.298998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f518738a300 x1874531145916544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697688.298999:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697688.299014:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145917312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697688.299017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145916544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697688.299090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697688.299111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697688.299122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f5187389500 x1874531145916672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697688.299133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145916672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697688.299159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697688.299178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697688.299187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f518738ad80 x1874531145916800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697688.299198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145916800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145916800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697688.299229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145916928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697688.299247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697688.299257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f518738aa00 x1874531145916928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697688.299279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145916928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145916928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697688.299301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145917056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697688.299321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697688.299332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f5187389180 x1874531145917056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697688.299344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145917056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145917056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697688.299387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145917184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697688.299404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697688.299414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f5187388a80 x1874531145917184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697688.299426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145917184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145917184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697688.299444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145917312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697688.299460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697688.299471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697688/real 1787697688] req@ffff8f5187388380 x1874531145917312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697688.299481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145917312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697688.299489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697688.299491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697688.299494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145917312:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697689.194602:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697690.218540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697691.242575:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697692.266558:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697693.290580:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697693.418428:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418442:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697693.418447:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418460:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418473:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697693.418480:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697693.418520:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145917440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697693.418547:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418551:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697693.418552:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418556:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418558:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697693.418561:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418569:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697693.418577:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145917568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697693.418589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418591:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697693.418592:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418595:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418597:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697693.418599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418602:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697693.418606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145917440:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697693.418612:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145917696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697693.418619:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697693.418622:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418624:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418626:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697693.418628:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418631:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697693.418639:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145917824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697693.418648:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697693.418653:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418657:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418663:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697693.418667:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697693.418673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418673:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697693.418689:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145917952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697693.418699:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697693.418704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418710:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418714:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697693.418719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f5187389f80 x1874531145917440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697693.418720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697693.418740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145917440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697693.418745:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145918080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697693.418752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.418760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697693.418762:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697693.418764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145917440:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697693.418769:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787697693.418772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697693.418773:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418780:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418786:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697693.418790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697693.418795:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697693.418800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f518738b100 x1874531145917568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697693.418801:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697693.418815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145918208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697693.418818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145917568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.418836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.418840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.418845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697693.418845:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418852:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697693.418854:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418857:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418860:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697693.418863:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697693.418873:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145918336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697693.418876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697693.418885:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697693.418888:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697693.418889:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697693.418892:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5593 00000100:00080000:3.0:1787697693.418895:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697693.418896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697693.418900:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697693.418904:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697693.418908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f5187388e00 x1874531145917696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697693.418913:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145918464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697693.418923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145917696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.418930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.418932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.418934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697693.418938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697693.418978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697693.418990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f518738b480 x1874531145917824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697693.419002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145917824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.419015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.419017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.419019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697693.419023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697693.419038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697693.419045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f5187388000 x1874531145917952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697693.419051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145917952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.419056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.419058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.419060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145917952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697693.419063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697693.419075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697693.419084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f518738a680 x1874531145918080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697693.419095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145918080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.419104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.419106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.419110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697693.419115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697693.419133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697693.419143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f518738b800 x1874531145918208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697693.419152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145918208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.419157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.419159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.419161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697693.419172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697693.419184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697693.419190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f518738b100 x1874531145918336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697693.419197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145918336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.419201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.419203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.419205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697693.419208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697693.419218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697693.419227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697693/real 1787697693] req@ffff8f5187389f80 x1874531145918464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697693.419240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145918464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697693.419247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697693.419250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697693.419253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145918464:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697694.314537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697695.338565:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697696.362513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697697.386581:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697698.410513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697698.538528:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538543:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697698.538548:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538558:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538570:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538577:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697698.538616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145918592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697698.538645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538648:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697698.538649:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538653:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538655:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538658:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697698.538667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697698.538676:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145918720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697698.538684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538686:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697698.538687:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538689:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538691:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538693:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697698.538695:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697698.538700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145918592:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697698.538704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145918848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697698.538716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697698.538722:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538726:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538729:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538733:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697698.538739:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697698.538752:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145918976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697698.538763:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538766:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697698.538768:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538773:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538776:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697698.538782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697698.538782:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697698.538790:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697698.538805:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145919104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697698.538815:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697698.538820:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538825:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538829:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538833:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697698.538839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183646d80 x1874531145918592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697698.538845:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697698.538873:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145919232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697698.538875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145918592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697698.538893:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538897:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697698.538898:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697698.538899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697698.538904:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538908:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697698.538909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697698.538913:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697698.538919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145918592:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697698.538919:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697698.538931:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145919360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697698.538933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145918720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697698.538944:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697698.538946:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697698.538947:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538950:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538952:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538955:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697698.538958:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697698.538964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697698.538967:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145919488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697698.538974:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697698.538978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183646a00 x1874531145918720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697698.538979:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697698.538982:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697698.538987:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5598 00000100:00080000:1.0:1787697698.538991:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697698.538996:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697698.539000:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697698.539006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145918720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697698.539007:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145919616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697698.539034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145918720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697698.539071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145918848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697698.539090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183647b80 x1874531145918848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145918848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145918848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697698.539126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145918976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697698.539145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183644380 x1874531145918976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145918976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145918976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697698.539176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145919104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697698.539187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183644700 x1874531145919104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145919104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145919104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697698.539210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697698.539219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183644000 x1874531145919232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145919232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697698.539246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697698.539254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183646680 x1874531145919360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145919360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697698.539276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697698.539284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183647480 x1874531145919488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145919488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697698.539305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697698.539314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697698.539319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697698/real 1787697698] req@ffff8f5183644e00 x1874531145919616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697698.539325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145919616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697698.539329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697698.539331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697698.539333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145919616:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697699.434603:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697700.458541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697701.482555:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697702.506522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697703.530589:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697703.658447:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697703.658458:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658466:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658475:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658481:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697703.658510:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145919744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697703.658531:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658535:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697703.658536:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658540:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658544:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658548:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658558:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697703.658572:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145919872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697703.658576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145919744:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697703.658589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658592:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697703.658594:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658599:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658602:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658606:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697703.658625:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145920000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697703.658631:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697703.658635:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658637:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658639:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658642:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697703.658652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145920128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697703.658656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697703.658664:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697703.658667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658673:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658675:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658678:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697703.658684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145920256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697703.658690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697703.658694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658696:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658698:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658700:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697703.658716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f5187388380 x1874531145919744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697703.658716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145920384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697703.658736:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658740:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697703.658742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658746:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658750:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697703.658754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145919744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697703.658756:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658764:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697703.658771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697703.658787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145920512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697703.658789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.658798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145919744:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697703.658800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697703.658807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658810:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658813:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658816:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658819:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697703.658827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145919872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697703.658828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145920640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697703.658846:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697703.658849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697703.658850:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697703.658853:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5603 00000100:00080000:3.0:1787697703.658855:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697703.658857:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697703.658860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697703.658868:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145920768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697703.658878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.658891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f5187388a80 x1874531145919872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.658909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145919872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.658915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.658917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.658919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145919872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697703.658923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145920000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697703.658937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.658943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f5187389180 x1874531145920000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.658950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145920000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.658954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.658956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.658958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145920000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697703.658961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145920128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697703.658971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.658976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f518738aa00 x1874531145920128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.658986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145920128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.658991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.658993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.658994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145920128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697703.658997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145920256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697703.659006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.659012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f518738ad80 x1874531145920256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.659018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145920256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.659023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.659027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.659030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145920256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697703.659034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697703.659051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.659061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f5187389500 x1874531145920384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.659073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145920384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.659083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.659085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.659087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697703.659091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697703.659104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.659110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f518738a300 x1874531145920512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.659116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145920512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.659121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.659123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.659125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697703.659137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697703.659146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.659151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f518738bb80 x1874531145920640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.659157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145920640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.659161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.659163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.659165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697703.659168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697703.659177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697703.659182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697703/real 1787697703] req@ffff8f5187389c00 x1874531145920768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697703.659188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145920768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697703.659192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697703.659194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697703.659195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145920768:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697704.554601:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697705.578575:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697706.602581:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697707.626561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697708.650605:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697708.778478:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697708.778492:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778502:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778514:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.778520:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697708.778567:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145920896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697708.778589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778592:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697708.778593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778596:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778598:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.778600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778609:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697708.778621:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145921024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697708.778632:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778635:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697708.778637:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697708.778643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145920896:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697708.778644:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778650:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.778656:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778661:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697708.778674:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145921152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697708.778685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778700:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697708.778702:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778705:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778717:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.778722:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697708.778727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697708.778745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145921280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697708.778755:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697708.778761:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778765:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778769:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.778773:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778779:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697708.778781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183645c00 x1874531145920896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697708.778802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145921408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697708.778806:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145920896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697708.778817:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778823:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697708.778827:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697708.778833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697708.778834:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:0.0:1787697708.778840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697708.778842:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697708.778844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145920896:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697708.778851:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778857:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697708.778871:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145921536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697708.778872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697708.778885:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778888:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697708.778889:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778893:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778896:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697708.778899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697708.778902:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778907:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697708.778914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183645180 x1874531145921024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697708.778917:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145921664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697708.778934:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778936:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697708.778938:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778943:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00100000:0.0:1787697708.778944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145921024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697708.778945:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.778951:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.778957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697708.778960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.778965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697708.778967:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145921792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697708.778971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697708.778983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921152:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697708.778985:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697708.778990:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697708.778992:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697708.778995:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5608 00000100:00080000:1.0:1787697708.778998:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697708.779002:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697708.779006:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697708.779013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697708.779019:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145921920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697708.779027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183647100 x1874531145921152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145921152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697708.779066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697708.779086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697708.779095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183647800 x1874531145921280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145921280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697708.779141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697708.779161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697708.779172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183644e00 x1874531145921408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145921408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145921408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697708.779212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697708.779226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697708.779233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183647480 x1874531145921536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145921536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697708.779256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697708.779265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697708.779271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183646680 x1874531145921664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145921664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697708.779287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697708.779295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697708.779299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183644000 x1874531145921792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145921792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697708.779317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697708.779326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697708.779330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697708/real 1787697708] req@ffff8f5183644700 x1874531145921920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697708.779336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145921920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697708.779342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697708.779345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697708.779348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145921920:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697709.674518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697710.698566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697711.722487:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697712.746531:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697713.770542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697713.898523:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898537:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697713.898541:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.898552:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.898562:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697713.898568:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697713.898622:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145922048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697713.898645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898648:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697713.898650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.898653:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.898656:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697713.898659:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697713.898667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697713.898678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145922176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697713.898685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697713.898691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145922048:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697713.898691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.898699:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.898702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697713.898704:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697713.898707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697713.898716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145922304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697713.898725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697713.898731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.898735:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.898740:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697713.898744:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697713.898750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697713.898753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697713.898770:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145922432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697713.898782:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697713.898787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f5187389c00 x1874531145922048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697713.898788:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697713.898792:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.898801:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.898807:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697713.898811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145922048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697713.898816:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697713.898826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697713.898826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697713.898837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697713.898843:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145922560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697713.898845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145922048:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697713.898864:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898867:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697713.898869:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.898875:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00100000:0.0:1787697713.898880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922176:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697713.898881:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697713.898890:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697713.898896:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697713.898909:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145922688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697713.898921:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898926:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697713.898929:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697713.898930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697713.898938:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.898943:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697713.898948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f518738bb80 x1874531145922176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697713.898950:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697713.898960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697713.898965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145922176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.898973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697713.898975:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145922816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697713.898979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.898989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922176:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697713.898993:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.898998:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697713.899002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922304:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697713.899003:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.899012:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.899016:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697713.899020:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697713.899027:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697713.899035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697713.899041:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145922944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697713.899052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f518738a300 x1874531145922304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697713.899054:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697713.899063:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697713.899068:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697713.899076:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5613 00000100:00080000:3.0:1787697713.899081:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697713.899086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145922304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697713.899087:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697713.899095:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697713.899104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697713.899122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697713.899127:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145923072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697713.899153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697713.899167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f5187389500 x1874531145922432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697713.899181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145922432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.899187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697713.899195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697713.899207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697713.899213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f518738ad80 x1874531145922560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697713.899220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145922560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.899225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145922560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697713.899241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145922688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697713.899252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697713.899258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f518738aa00 x1874531145922688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697713.899265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145922688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.899275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145922688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697713.899283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145922816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697713.899296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697713.899303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f5187389180 x1874531145922816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697713.899310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145922816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.899316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145922816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697713.899323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145922944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697713.899332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697713.899338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f5187388a80 x1874531145922944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697713.899344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145922944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.899349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145922944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697713.899406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145923072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697713.899428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697713.899440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff8f5187388380 x1874531145923072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697713.899452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145923072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697713.899460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697713.899463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697713.899466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145923072:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697714.794544:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697715.818509:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697716.842542:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697717.866494:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697718.890573:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697719.018437:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018448:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697719.018452:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018461:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018470:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697719.018476:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697719.018508:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145923200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697719.018558:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018561:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697719.018564:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018570:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018574:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697719.018579:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018589:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697719.018603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145923328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697719.018605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145923200:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697719.018613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697719.018618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018622:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018625:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697719.018628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697719.018640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145923456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697719.018648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697719.018652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018655:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018658:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697719.018661:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697719.018663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697719.018676:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145923584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697719.018685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697719.018689:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018693:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018696:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697719.018699:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697719.018704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183644380 x1874531145923200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697719.018712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145923712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697719.018720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145923200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697719.018721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018724:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697719.018727:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018730:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:0.0:1787697719.018731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697719.018733:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697719.018735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697719.018736:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697719.018739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145923200:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697719.018741:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697719.018749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145923840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697719.018757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697719.018760:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697719.018761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697719.018762:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018766:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018769:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697719.018772:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018777:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697719.018787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145923968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697719.018797:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697719.018799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697719.018800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697719.018802:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697719.018806:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018809:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697719.018812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183647b80 x1874531145923328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697719.018812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697719.018825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145923328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697719.018826:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145924096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697719.018835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697719.018835:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697719.018838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697719.018839:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697719.018840:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697719.018842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697719.018844:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5618 00000100:00080000:1.0:1787697719.018847:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697719.018848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923456:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697719.018851:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697719.018855:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697719.018866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.018877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183646a00 x1874531145923456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697719.018879:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145924224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697719.018890:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145923456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.018899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.018902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.018905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697719.018912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697719.018933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.018943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183646d80 x1874531145923584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697719.018956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145923584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.018961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.018964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.018966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697719.018969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697719.018981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.018987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183644700 x1874531145923712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697719.018995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145923712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.019000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.019001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.019003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145923712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697719.019007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145923840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697719.019016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.019022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183644000 x1874531145923840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697719.019029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145923840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.019039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.019041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.019043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145923840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697719.019055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145923968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697719.019065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.019071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183646680 x1874531145923968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697719.019077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145923968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.019083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.019084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.019086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145923968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697719.019089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145924096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697719.019098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.019104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183647480 x1874531145924096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697719.019111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145924096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.019115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.019117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.019119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145924096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697719.019123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145924224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697719.019132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697719.019137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697719/real 1787697719] req@ffff8f5183644e00 x1874531145924224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697719.019144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145924224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697719.019149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697719.019150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697719.019152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145924224:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697719.914542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697720.938615:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697721.962611:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697722.986580:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697724.010643:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697724.138519:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138531:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697724.138535:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138547:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138560:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.138567:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697724.138613:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145924352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697724.138644:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138648:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697724.138650:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138655:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138659:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.138663:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697724.138674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697724.138687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145924480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697724.138692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145924352:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697724.138701:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697724.138708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138712:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138717:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.138720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697724.138725:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697724.138737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145924608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697724.138746:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697724.138751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138754:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138766:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.138772:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697724.138777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787697724.138780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697724.138792:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145924736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697724.138808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697724.138812:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138817:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138821:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.138824:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697724.138829:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697724.138837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace91c00 x1874531145924352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697724.138843:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145924864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697724.138861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138865:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697724.138867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697724.138872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145924352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697724.138875:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138881:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.138888:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697724.138891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697724.138893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697724.138899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697724.138906:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145924992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697724.138907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145924352:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697724.138922:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138926:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697724.138928:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138932:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138936:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697724.138941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697724.138942:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697724.138950:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697724.138962:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145925120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697724.138972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697724.138975:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.138979:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697724.138982:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.138987:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5623 00000100:00080000:0.0:1787697724.138992:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697724.138992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace91880 x1874531145924480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697724.139000:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697724.139008:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697724.139015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145924480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697724.139022:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145925248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697724.139026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697724.139039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697724.139049:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697724.139053:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697724.139055:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697724.139061:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5623 00000800:00000100:1.0:1787697724.139065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697724.139067:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697724.139074:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace91500 x1874531145924608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697724.139082:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697724.139088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145924608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697724.139097:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145925376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697724.139098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697724.139113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697724.139132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace92300 x1874531145924736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697724.139161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145924736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697724.139182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697724.139197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace93480 x1874531145924864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697724.139216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145924864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145924864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697724.139235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145924992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697724.139254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace92d80 x1874531145924992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697724.139277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145924992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145924992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697724.139310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145925120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697724.139329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace91c00 x1874531145925120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697724.139351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145925120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145925120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697724.139394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145925248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697724.139413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace93100 x1874531145925248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697724.139447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145925248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145925248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697724.139467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145925376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697724.139481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697724.139488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697724/real 1787697724] req@ffff8f51ace92680 x1874531145925376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697724.139495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145925376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697724.139500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697724.139502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697724.139505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145925376:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697725.034558:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697726.058640:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697727.082559:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697728.106594:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697729.130512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697729.258463:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258472:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697729.258475:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697729.258491:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258499:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258504:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697729.258532:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145925504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697729.258549:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697729.258554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697729.258559:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258562:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258565:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697729.258574:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697729.258584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145925504:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697729.258585:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145925632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697729.258599:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258602:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697729.258603:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697729.258606:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258609:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258612:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697729.258615:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697729.258631:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145925760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697729.258638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697729.258643:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697729.258645:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5628 00000800:00000100:1.0:1787697729.258648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697729.258648:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258651:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697729.258655:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697729.258663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145925888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697729.258670:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697729.258674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697729.258678:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258681:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697729.258683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f5187389f80 x1874531145925504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697729.258684:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697729.258688:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697729.258696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145926016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697729.258700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145925504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697729.258704:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697729.258709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697729.258709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697729.258713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697729.258713:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258716:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697729.258717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145925504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697729.258719:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697729.258723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697729.258731:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145926144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697729.258736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145925632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697729.258738:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697729.258743:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697729.258746:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258749:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258752:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787697729.258754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697729.258756:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697729.258764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145926272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697729.258765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f518738b100 x1874531145925632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.258777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145925632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697729.258779:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697729.258784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697729.258784:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697729.258787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697729.258788:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00100000:1.0:1787697729.258790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145925632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697729.258791:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258794:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697729.258795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145925760:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697729.258798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697729.258806:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145926400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697729.258812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697729.258814:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697729.258817:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697729.258818:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697729.258821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f518738b800 x1874531145925760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697729.258821:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5628 00000100:00080000:3.0:1787697729.258824:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697729.258827:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697729.258831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145925760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697729.258831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697729.258838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697729.258839:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145926528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697729.258841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.258844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145925760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697729.258849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145925888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697729.258876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697729.258887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f518738a680 x1874531145925888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.258898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145925888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697729.258905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697729.258908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.258910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145925888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697729.258915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145926016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697729.258932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697729.258942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f5187388000 x1874531145926016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.258958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145926016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697729.258965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697729.258968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.258971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145926016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697729.258984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697729.259001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697729.259012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f518738b480 x1874531145926144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.259021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145926144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697729.259026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697729.259028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.259030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697729.259033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697729.259045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697729.259050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f5187388e00 x1874531145926272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.259057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145926272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697729.259064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697729.259066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.259068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697729.259071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697729.259079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697729.259083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f5187389f80 x1874531145926400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.259089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145926400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697729.259094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697729.259095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.259097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697729.259100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697729.259108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697729.259112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697729/real 1787697729] req@ffff8f5187388380 x1874531145926528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697729.259119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145926528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697729.259123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697729.259124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697729.259126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145926528:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697730.154577:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697731.178570:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697732.202532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697733.226579:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697734.250498:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697734.378503:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.378517:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697734.378522:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.378535:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378548:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697734.378557:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697734.378603:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145926656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697734.378635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.378640:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697734.378642:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.378647:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378651:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697734.378655:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.378667:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697734.378681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145926656:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697734.378687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145926784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697734.378700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.378712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697734.378715:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.378720:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378725:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697734.378729:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.378735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697734.378747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531145926912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697734.378755:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.378759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787697734.378762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697734.378763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.378771:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378774:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697734.378778:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.378784:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697734.378795:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531145927040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697734.378803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.378806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697734.378808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.378813:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378817:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697734.378821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace90e00 x1874531145926656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697734.378824:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.378834:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697734.378846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145926656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697734.378848:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531145927168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697734.378861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697734.378866:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697734.378866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697734.378873:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787697734.378874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145926656:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697734.378879:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.378885:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378888:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697734.378891:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.378896:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697734.378900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145926784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697734.378908:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145927296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697734.378919:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.378923:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697734.378925:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697734.378928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697734.378932:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.378938:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697734.378942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace90380 x1874531145926784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697734.378946:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697734.378958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145926784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697734.378964:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697734.378965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.378968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.378973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145926784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697734.378980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145926912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697734.378986:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145927424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697734.378997:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.379001:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697734.379004:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697734.379005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697734.379011:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5634 00000100:00080000:0.0:1787697734.379018:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697734.379023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace93b80 x1874531145926912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697734.379025:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.379035:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697734.379040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145926912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697734.379052:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145927552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697734.379054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145926912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697734.379063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145927040:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787697734.379072:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697734.379076:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697734.379079:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697734.379085:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5634 00000800:00000100:1.0:1787697734.379085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697734.379093:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697734.379098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace93800 x1874531145927040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697734.379099:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697734.379109:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697734.379112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145927040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.379127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145927040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697734.379129:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145927680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697734.379133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145927168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697734.379160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697734.379183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace90000 x1874531145927168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697734.379201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145927168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.379215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145927168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697734.379232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697734.379266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697734.379278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace90700 x1874531145927296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697734.379290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145927296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.379306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697734.379312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697734.379330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697734.379340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace91180 x1874531145927424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697734.379352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145927424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.379397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697734.379403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697734.379422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697734.379431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace90e00 x1874531145927552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697734.379442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145927552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.379456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697734.379462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697734.379479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697734.379487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697734/real 1787697734] req@ffff8f51ace92680 x1874531145927680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697734.379494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145927680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697734.379499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697734.379501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697734.379503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145927680:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697735.274523:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697736.298545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697737.322489:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697738.346543:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697739.370587:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697739.498490:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.498499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697739.498503:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498513:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.498525:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498541:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697739.498591:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145927808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697739.498627:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.498631:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697739.498633:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498638:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.498642:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498646:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697739.498658:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697739.498671:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145927936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697739.498681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145927808:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697739.498685:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.498690:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697739.498692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498711:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.498719:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498724:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697739.498730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697739.498746:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145928064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697739.498756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.498759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697739.498761:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498765:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5639 00000800:00000100:1.0:1787697739.498770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697739.498771:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498777:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697739.498781:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697739.498793:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145928192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697739.498803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.498806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697739.498808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498812:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.498816:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498821:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697739.498826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f5187388a80 x1874531145927808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697739.498827:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697739.498844:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145928320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697739.498850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145927808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.498860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.498865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697739.498868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697739.498869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145927808:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697739.498877:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697739.498880:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498885:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.498888:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498891:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697739.498896:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697739.498897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145927936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697739.498911:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145928448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697739.498920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697739.498922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697739.498924:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697739.498927:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498931:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.498933:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697739.498936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f5187389180 x1874531145927936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697739.498939:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697739.498945:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697739.498951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145927936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697739.498955:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145928576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697739.498964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697739.498969:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.498972:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697739.498974:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697739.498977:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:1.0:1787697739.498978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697739.498978:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.498984:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697739.498985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145927936:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697739.498988:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697739.498993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145928064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697739.498998:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145928704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697739.499006:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697739.499009:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697739.499010:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697739.499013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697739.499014:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5639 00000100:00080000:3.0:1787697739.499020:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697739.499023:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697739.499025:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697739.499027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f518738aa00 x1874531145928064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697739.499034:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145928832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697739.499040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145928064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145928064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697739.499057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145928192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697739.499069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697739.499076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f518738ad80 x1874531145928192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697739.499082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145928192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145928192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697739.499092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145928320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697739.499106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697739.499111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f5187389500 x1874531145928320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697739.499123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145928320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145928320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697739.499141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697739.499150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697739.499155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f518738a300 x1874531145928448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697739.499175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145928448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697739.499188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697739.499201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697739.499208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f518738bb80 x1874531145928576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697739.499215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145928576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697739.499236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697739.499253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697739.499264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f5187389c00 x1874531145928704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697739.499282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145928704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697739.499294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697739.499307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697739.499313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697739/real 1787697739] req@ffff8f5187388a80 x1874531145928832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697739.499320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145928832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697739.499325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697739.499327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697739.499329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145928832:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697740.394566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697741.418566:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697742.442506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697743.466530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697744.490488:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697744.618451:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618458:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697744.618461:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697744.618469:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618477:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618482:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697744.618511:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145928960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697744.618534:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618537:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697744.618539:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697744.618544:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618548:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618551:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618560:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697744.618573:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145929088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697744.618581:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697744.618584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145928960:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697744.618584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697744.618587:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697744.618592:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618595:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618600:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697744.618609:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145929216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697744.618615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618624:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697744.618625:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697744.618628:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618630:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618632:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618636:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697744.618644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697744.618644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145929344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697744.618653:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618655:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697744.618656:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697744.618658:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618661:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618664:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618668:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697744.618678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145929472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697744.618687:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697744.618692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697744.618695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f5187388380 x1874531145928960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697744.618699:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618711:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697744.618722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145928960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697744.618727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145929600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697744.618744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697744.618746:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697744.618753:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697744.618757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697744.618762:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00100000:1.0:1787697744.618766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145928960:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697744.618768:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618776:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618781:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697744.618790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697744.618794:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145929728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697744.618813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697744.618815:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697744.618820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697744.618822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697744.618828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f5187389f80 x1874531145929088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697744.618828:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618833:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618839:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618846:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697744.618851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145929088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697744.618860:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145929856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697744.618862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.618872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697744.618877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697744.618878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929088:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697744.618883:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697744.618887:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697744.618891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697744.618892:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5644 00000100:00080000:3.0:1787697744.618897:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697744.618904:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697744.618908:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697744.618919:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145929984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697744.618920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.618934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f5187388e00 x1874531145929216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.618948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145929216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.618956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.618959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.618962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697744.618968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697744.618984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.618993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f518738b480 x1874531145929344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.619004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145929344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.619012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.619015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.619017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697744.619022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697744.619036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.619044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f5187388000 x1874531145929472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.619060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145929472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.619068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.619071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.619100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145929472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697744.619106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697744.619122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.619132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f518738a680 x1874531145929600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.619143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145929600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.619151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.619153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.619156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697744.619172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697744.619185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.619193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f518738b800 x1874531145929728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.619204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145929728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.619216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.619219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.619222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697744.619227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697744.619240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.619247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f518738b100 x1874531145929856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.619257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145929856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.619264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.619267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.619270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697744.619275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697744.619290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697744.619298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697744/real 1787697744] req@ffff8f5187388380 x1874531145929984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697744.619309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145929984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697744.619316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697744.619319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697744.619322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145929984:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697745.514544:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697746.538504:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697747.562580:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697748.586529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697749.610533:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697749.738485:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738492:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697749.738495:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738503:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738514:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738520:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697749.738556:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145930112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738577:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738579:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697749.738580:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738583:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738586:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738589:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697749.738596:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697749.738606:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145930240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738612:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738615:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697749.738616:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738619:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738621:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738625:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697749.738629:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697749.738641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145930112:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697749.738642:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145930368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697749.738669:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738679:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697749.738687:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697749.738698:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145930496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697749.738709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738713:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738729:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738732:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697749.738735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697749.738742:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145930624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738748:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738751:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697749.738752:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738754:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738756:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738758:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697749.738761:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697749.738769:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145930752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738779:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738781:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697749.738784:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738787:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738790:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738797:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697749.738797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930240:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787697749.738810:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697749.738828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145930880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738841:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697749.738843:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697749.738847:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738851:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738856:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697749.738862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930368:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787697749.738866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697749.738885:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145931008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697749.738897:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697749.738900:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697749.738902:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697749.738908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930496:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787697749.738911:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5649 00000100:00080000:0.0:1787697749.738923:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697749.738929:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697749.738933:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697749.738945:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145931136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697749.738960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697749.739007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145930752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697749.739032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145930880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697749.739059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145931008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697749.739081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145931136:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697750.634522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697751.658604:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697752.682580:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787697752.810610:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787697752.810644:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787697752.810656:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697752.810699:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787697752.810713:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697752.810723:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697752.810733:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787697752.810744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace90e00 x1874531145931136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697752.810751:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697752.810764:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787697752.810780:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787697752.810781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145931136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697752.810792:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787697752.810797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.810805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.810810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145931136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697752.810820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace92d80 x1874531145930368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.810834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145930368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.810842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.810846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.810849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697752.810856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace93480 x1874531145930496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.810867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145930496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.810875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.810880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.810883:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697752.810890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace92300 x1874531145930624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.810901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145930624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.810908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.810911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.810915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697752.810922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace91500 x1874531145930752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.810982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145930752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.810991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.810995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.810998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145930752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697752.811005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace91880 x1874531145930880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.811018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145930880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.811025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.811029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.811032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145930880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697752.811037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace93100 x1874531145930112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.811064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145930112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.811071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.811074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.811077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145930112:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697752.811084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace91c00 x1874531145930240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.811095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145930240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.811102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.811106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.811109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145930240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697752.811139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697749/real 1787697752] req@ffff8f51ace92680 x1874531145931008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697752.811150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145931008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697752.811157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697752.811159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697752.811162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145931008:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697753.706507:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697754.730517:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697754.858495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697754.858502:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697754.858505:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697754.858513:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858520:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858526:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697754.858555:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145931264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697754.858582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697754.858585:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697754.858587:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697754.858590:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858596:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858606:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697754.858621:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145931392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697754.858633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145931264:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697754.858634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697754.858644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697754.858646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697754.858660:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858664:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858668:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697754.858686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145931520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697754.858696:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697754.858697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697754.858702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697754.858705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697754.858710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858714:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858719:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697754.858737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145931648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697754.858742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f5187388a80 x1874531145931264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697754.858758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697754.858763:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697754.858765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697754.858772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145931264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697754.858772:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858778:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858784:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697754.858806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697754.858809:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145931776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697754.858814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697754.858825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697754.858829:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697754.858830:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697754.858832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145931264:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697754.858844:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858848:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858852:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858857:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697754.858861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697754.858872:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145931904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697754.858882:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697754.858886:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697754.858888:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697754.858890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697754.858913:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:3.0:1787697754.858917:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.858920:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858925:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697754.858934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f5187389c00 x1874531145931392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697754.858939:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145932032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697754.858955:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697754.858958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145931392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697754.858960:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697754.858963:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697754.858973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697754.858973:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00080000:1.0:1787697754.858980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697754.858980:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697754.858986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931392:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697754.858988:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.858997:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697754.858998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697754.859013:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145932160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697754.859023:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697754.859026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697754.859029:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697754.859033:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697754.859037:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5654 00000100:00100000:1.0:1787697754.859042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f518738bb80 x1874531145931520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697754.859043:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697754.859051:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697754.859061:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697754.859065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145931520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697754.859076:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145932288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697754.859080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697754.859097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697754.859118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697754.859130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f518738a300 x1874531145931648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.859140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145931648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.859150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697754.859158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697754.859176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697754.859187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f5187389500 x1874531145931776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.859204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145931776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.859215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145931776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697754.859235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145931904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697754.859255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697754.859266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f518738ad80 x1874531145931904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.859278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145931904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.859289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145931904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697754.859297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145932032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697754.859315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697754.859325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f518738aa00 x1874531145932032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.859337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145932032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.859347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145932032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697754.859384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145932160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697754.859403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697754.859413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f5187389180 x1874531145932160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.859425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145932160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.859442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145932160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697754.859451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145932288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697754.859468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697754.859479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff8f5187388a80 x1874531145932288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.859489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145932288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.859496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.859499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697754.859502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145932288:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697755.754505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697756.778483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697757.802494:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697758.826491:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697759.850499:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697759.978433:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978442:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697759.978447:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978456:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:0.0:1787697759.978466:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978473:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697759.978508:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145932416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697759.978534:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978537:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697759.978539:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978543:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:0.0:1787697759.978547:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978551:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697759.978560:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697759.978573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145932416:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697759.978575:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145932544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697759.978595:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978598:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697759.978600:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978604:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:0.0:1787697759.978607:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978611:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697759.978615:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697759.978626:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531145932672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697759.978636:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697759.978641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697759.978643:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697759.978647:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978652:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:0.0:1787697759.978657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978661:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697759.978666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697759.978678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531145932800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697759.978684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace91180 x1874531145932416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.978691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697759.978704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00100000:1.0:1787697759.978714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145932416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.978716:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978724:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697759.978731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697759.978732:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697759.978739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.978745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145932416:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697759.978748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531145932928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697759.978759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697759.978764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978768:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00100000:1.0:1787697759.978772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932544:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697759.978774:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978782:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697759.978786:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697759.978800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697759.978802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145933056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697759.978841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace90700 x1874531145932544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.978853:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697759.978857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145932544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.978859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697759.978863:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.978873:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:1.0:1787697759.978876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697759.978878:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697759.978883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697759.978887:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697759.978891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932544:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787697759.978894:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697759.978900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697759.978906:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145933184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697759.978917:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978919:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697759.978921:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697759.978925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697759.978927:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:0.0:1787697759.978932:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978935:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697759.978939:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697759.978944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace90000 x1874531145932672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697759.978948:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145933312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697759.978969:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.978972:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697759.978974:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697759.978975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145932672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.978978:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5659 00000100:00080000:0.0:1787697759.978983:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697759.978988:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697759.978994:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697759.978996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697759.979001:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145933440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697759.979002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697759.979021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697759.979040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697759.979051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace93800 x1874531145932800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.979061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145932800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.979068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.979071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697759.979079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697759.979093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697759.979101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace93b80 x1874531145932928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.979111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145932928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.979119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.979122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145932928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697759.979138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697759.979152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697759.979161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace90380 x1874531145933056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.979170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145933056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.979178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.979181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697759.979189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697759.979203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697759.979211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace91180 x1874531145933184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.979221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145933184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.979228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.979231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697759.979239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697759.979266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697759.979276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace92680 x1874531145933312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.979286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145933312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.979293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.979296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697759.979304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697759.979319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697759.979327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff8f51ace91c00 x1874531145933440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.979337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145933440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.979344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.979346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697759.979349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145933440:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697760.874528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697761.898478:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697762.922534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697763.946519:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697764.970524:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697765.098476:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098486:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697765.098490:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697765.098499:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098509:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697765.098516:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697765.098547:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145933568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697765.098574:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098577:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697765.098579:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697765.098590:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697765.098598:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098606:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697765.098620:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145933696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697765.098627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145933568:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697765.098634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098639:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697765.098641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697765.098645:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098648:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697765.098652:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098658:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697765.098670:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145933824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697765.098677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098679:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697765.098680:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697765.098683:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098687:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697765.098688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697765.098691:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098696:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697765.098703:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145933952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697765.098709:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098711:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697765.098712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697765.098721:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098733:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697765.098737:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098741:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697765.098745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f5187388380 x1874531145933568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697765.098752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145934080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697765.098763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697765.098767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697765.098771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145933568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697765.098773:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098778:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697765.098783:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098791:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697765.098793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.098800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697765.098801:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145934208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697765.098805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145933568:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697765.098819:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098823:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697765.098825:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697765.098830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145933696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697765.098831:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098837:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697765.098844:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098848:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697765.098859:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145934336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697765.098866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697765.098870:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697765.098873:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697765.098875:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697765.098878:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098881:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697765.098882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f518738b100 x1874531145933696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697765.098886:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098892:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697765.098903:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145934464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697765.098906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145933696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697765.098927:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697765.098928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.098935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697765.098935:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697765.098939:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697765.098942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145933696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697765.098947:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5664 00000100:00080000:3.0:1787697765.098951:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697765.098952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145933824:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697765.098956:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697765.098961:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697765.098971:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145934592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697765.098977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.098993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f518738b800 x1874531145933824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145933824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145933824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697765.099024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145933952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697765.099039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.099047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f518738a680 x1874531145933952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145933952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145933952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697765.099083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145934080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697765.099096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.099105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f5187388000 x1874531145934080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145934080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145934080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697765.099133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697765.099147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.099157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f518738b480 x1874531145934208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145934208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697765.099204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697765.099224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.099236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f5187388e00 x1874531145934336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145934336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697765.099265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697765.099282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.099292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f5187389f80 x1874531145934464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145934464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697765.099322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697765.099338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697765.099349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697765/real 1787697765] req@ffff8f5187388380 x1874531145934592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697765.099380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145934592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697765.099387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697765.099389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697765.099392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145934592:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697765.994495:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697767.018520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697768.042504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697769.066527:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697770.090525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697770.218441:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218450:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697770.218454:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218463:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218473:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218479:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697770.218509:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145934720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697770.218532:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218535:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697770.218537:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218541:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218545:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218548:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218557:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697770.218568:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145934848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697770.218571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145934720:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697770.218589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218596:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697770.218598:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218602:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218606:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218609:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218614:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697770.218625:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145934976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697770.218635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697770.218654:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697770.218660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218665:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218668:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218672:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218677:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697770.218687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace93100 x1874531145934720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697770.218695:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145935104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697770.218706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697770.218708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145934720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697770.218713:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697770.218716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218725:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:1.0:1787697770.218725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697770.218730:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697770.218734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.218738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145934720:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697770.218743:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218749:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697770.218760:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145935232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697770.218767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145934848:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697770.218772:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697770.218781:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218786:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218790:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218795:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218801:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697770.218802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697770.218815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145935360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697770.218821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace91880 x1874531145934848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697770.218828:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218834:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697770.218836:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218840:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218844:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218851:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218856:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697770.218858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145934848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.218866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697770.218869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145935488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697770.218871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.218877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145934848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697770.218887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145934976:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697770.218891:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218896:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697770.218898:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218903:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5669 00000800:00000100:1.0:1787697770.218907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697770.218908:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218915:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218920:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697770.218922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace91500 x1874531145934976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697770.218937:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145935616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697770.218937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145934976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.218943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.218947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.218954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145934976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697770.218966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145935104:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787697770.218974:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697770.218978:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697770.218979:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697770.218984:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5669 00000100:00080000:0.0:1787697770.218987:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697770.218990:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697770.218995:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697770.219005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697770.219007:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145935744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697770.219018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace92300 x1874531145935104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697770.219031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145935104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.219037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.219039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.219041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145935104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697770.219051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145935232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697770.219067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697770.219085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace93480 x1874531145935232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697770.219096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145935232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.219103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.219106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.219109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145935232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697770.219114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697770.219138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697770.219145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace92d80 x1874531145935360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697770.219152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145935360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.219161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.219162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.219164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697770.219167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697770.219176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697770.219182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace90e00 x1874531145935488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697770.219193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145935488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.219199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.219202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.219205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697770.219210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697770.219227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697770.219237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace93100 x1874531145935616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697770.219245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145935616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.219249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.219251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.219253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697770.219257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697770.219268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697770.219273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697770/real 1787697770] req@ffff8f51ace91c00 x1874531145935744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697770.219279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145935744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697770.219284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697770.219285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697770.219287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145935744:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697771.114509:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697772.138503:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697773.162493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697774.186497:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697775.210514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697775.338470:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338481:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697775.338485:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338496:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338507:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338514:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697775.338559:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531145935872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338587:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338591:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697775.338593:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338598:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338601:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338606:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338615:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697775.338626:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531145936000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338633:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338635:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697775.338637:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697775.338641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145935872:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697775.338642:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338645:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338650:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697775.338660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531145936128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338666:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338668:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697775.338669:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338671:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338673:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338675:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338678:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697775.338683:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531145936256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697775.338692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338695:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338697:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338699:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697775.338708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697775.338709:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531145936384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338715:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338717:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697775.338718:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338721:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338724:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338727:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697775.338734:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531145936512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338739:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697775.338742:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338744:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338746:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338749:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697775.338757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace92680 x1874531145935872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697775.338760:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531145936640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338782:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697775.338787:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697775.338790:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338792:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338794:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697775.338797:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697775.338803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531145935872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697775.338805:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145936768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697775.338823:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697775.338827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697775.338829:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697775.338831:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697775.338832:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5674 00000100:00080000:0.0:1787697775.338836:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697775.338840:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697775.338841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787697775.338843:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697775.338850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145935872:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697775.338852:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145936896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697775.338879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697775.338904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.338917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace91180 x1874531145936000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.338928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531145936000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.338934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.338936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.338938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697775.338941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697775.338954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.338961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace90380 x1874531145936128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.338972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531145936128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.338979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.338982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.338985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697775.338990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697775.339008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.339019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace93b80 x1874531145936256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.339036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531145936256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.339044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.339047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.339049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697775.339055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697775.339070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.339076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace93800 x1874531145936384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.339083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531145936384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.339088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.339090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.339092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145936384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697775.339095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697775.339103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.339108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace90000 x1874531145936512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.339114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531145936512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.339122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.339124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.339126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697775.339128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697775.339136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.339141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace90700 x1874531145936640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.339147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531145936640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.339151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.339152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.339154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697775.339157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697775.339165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.339169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace91c00 x1874531145936768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.339184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145936768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.339188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.339190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.339192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697775.339195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697775.339205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697775.339210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697775/real 1787697775] req@ffff8f51ace93100 x1874531145936896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697775.339220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145936896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697775.339228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697775.339230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697775.339233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145936896:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697776.234509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697777.258564:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697778.282478:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697779.306562:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697780.330554:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697780.458507:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.458517:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697780.458521:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.458530:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:3.0:1787697780.458541:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697780.458547:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697780.458584:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145937024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697780.458607:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.458611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697780.458613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.458617:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:3.0:1787697780.458620:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697780.458624:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697780.458633:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697780.458645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145937024:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697780.458645:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145937152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697780.458657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.458658:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697780.458659:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.458662:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:3.0:1787697780.458665:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697780.458668:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697780.458673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697780.458686:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145937280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697780.458701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697780.458703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697780.458708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697780.458710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.458716:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:3.0:1787697780.458720:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697780.458724:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697780.458729:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697780.458741:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145937408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697780.458746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f5187388a80 x1874531145937024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697780.458761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.458767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697780.458769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697780.458770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145937024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697780.458777:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:1.0:1787697780.458783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697780.458784:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697780.458787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697780.458792:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697780.458793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145937024:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697780.458802:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697780.458818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697780.458819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145937536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697780.458852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697780.458854:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.458859:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697780.458861:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.458867:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00100000:1.0:1787697780.458869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f5187389180 x1874531145937152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697780.458874:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697780.458880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145937152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697780.458881:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697780.458889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697780.458892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697780.458898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.458901:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697780.458906:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145937664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697780.458912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697780.458923:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.458927:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697780.458929:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.458933:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5680 00000800:00000100:1.0:1787697780.458937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697780.458940:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697780.458946:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697780.458949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f518738aa00 x1874531145937280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697780.458952:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697780.458960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145937280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697780.458966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697780.458969:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145937792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697780.458971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.458978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697780.458987:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697780.458990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937408:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697780.458992:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697780.458997:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697780.459002:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:3.0:1787697780.459006:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697780.459010:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697780.459015:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697780.459016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697780.459027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f518738ad80 x1874531145937408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697780.459031:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145937920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697780.459062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145937408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697780.459062:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697780.459069:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697780.459073:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697780.459080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697780.459081:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5680 00000100:00080000:1.0:1787697780.459086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697780.459088:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697780.459089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937408:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697780.459096:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697780.459102:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697780.459114:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145938048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697780.459115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697780.459135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697780.459143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f5187389500 x1874531145937536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697780.459152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145937536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697780.459157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697780.459158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.459160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145937536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697780.459163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145937664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697780.459172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697780.459177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f518738a300 x1874531145937664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697780.459183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145937664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697780.459192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697780.459194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.459195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145937664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697780.459198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145937792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697780.459206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697780.459211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f518738bb80 x1874531145937792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697780.459217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145937792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697780.459221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697780.459222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.459224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145937792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697780.459226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145937920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697780.459234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697780.459239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f5187389c00 x1874531145937920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697780.459244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145937920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697780.459248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697780.459250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.459252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145937920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697780.459259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145938048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697780.459267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697780.459272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697780/real 1787697780] req@ffff8f5187388a80 x1874531145938048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697780.459279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145938048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697780.459283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697780.459285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697780.459286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145938048:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697781.354534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697782.378551:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697783.402587:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697784.426604:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697785.450578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697785.578527:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.578539:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697785.578543:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697785.578555:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578569:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.578577:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697785.578622:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145938176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697785.578653:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.578657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697785.578659:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697785.578665:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578669:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.578672:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697785.578683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697785.578695:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145938304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697785.578705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.578708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697785.578710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697785.578716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145938176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697785.578717:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.578728:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697785.578732:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697785.578744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145938432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697785.578752:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.578756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697785.578758:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697785.578762:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578765:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.578769:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697785.578774:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697785.578782:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145938560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697785.578798:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697785.578800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697785.578803:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697785.578806:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697785.578811:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578815:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.578819:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697785.578824:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697785.578836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145938688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697785.578844:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.578848:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697785.578850:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697785.578854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f5187388380 x1874531145938176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697785.578858:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578865:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.578870:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697785.578875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697785.578877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145938176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.578886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697785.578888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145938816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697785.578891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.578897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145938176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697785.578907:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.578911:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697785.578913:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697785.578917:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578922:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697785.578923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938304:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697785.578928:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697785.578935:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697785.578947:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145938944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697785.578951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697785.578962:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697785.578963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f5187389f80 x1874531145938304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697785.578971:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697785.578975:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697785.578979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145938304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697785.578983:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:3.0:1787697785.578989:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697785.578991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.578996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697785.578996:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938304:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697785.579005:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697785.579007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697785.579021:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145939072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697785.579026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f5187388e00 x1874531145938432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697785.579036:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697785.579045:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697785.579049:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697785.579051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145938432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697785.579056:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5685 00000100:00080000:1.0:1787697785.579061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697785.579063:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697785.579070:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697785.579071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697785.579075:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697785.579077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697785.579084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697785.579091:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145939200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697785.579099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f518738b480 x1874531145938560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697785.579121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145938560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.579126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.579128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.579131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697785.579135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697785.579145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f5187388000 x1874531145938688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697785.579158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145938688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.579163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.579165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.579166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145938688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697785.579169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145938816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697785.579178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f518738a680 x1874531145938816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697785.579190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145938816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.579199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.579201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.579203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145938816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697785.579212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145938944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697785.579221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f518738b800 x1874531145938944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697785.579233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145938944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.579237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.579239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.579240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145938944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697785.579243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145939072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697785.579252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f518738b100 x1874531145939072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697785.579263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145939072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.579267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.579268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.579271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145939072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697785.579274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145939200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697785.579282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697785.579287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697785/real 1787697785] req@ffff8f5187388380 x1874531145939200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697785.579293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145939200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697785.579297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697785.579299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697785.579300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145939200:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697786.474527:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697787.498554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697788.522609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697789.546572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697790.570568:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697790.698438:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698445:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697790.698448:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698455:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698463:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698468:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697790.698495:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145939328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698515:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698517:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697790.698518:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698521:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698523:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698525:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697790.698531:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697790.698539:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145939456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698546:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698549:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697790.698550:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698552:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698556:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698559:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697790.698562:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697790.698570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145939328:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697790.698571:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145939584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697790.698592:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698596:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698598:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698600:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697790.698603:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697790.698610:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145939712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697790.698619:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698622:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698627:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698629:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697790.698632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697790.698637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145939840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697790.698651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697790.698657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697790.698661:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698671:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698674:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697790.698678:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697790.698689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145939968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698696:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698698:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697790.698699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698700:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698704:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697790.698706:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697790.698711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145940096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698718:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697790.698720:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698723:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698725:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698729:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697790.698736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f5187388a80 x1874531145939328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697790.698736:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697790.698749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145940224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697790.698761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697790.698765:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697790.698766:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697790.698771:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5690 00000100:00080000:3.0:1787697790.698774:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697790.698779:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697790.698780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145939328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697790.698784:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697790.698795:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145940352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697790.698837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.698843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.698848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145939328:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697790.698860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697790.698882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.698891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f5187389c00 x1874531145939456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.698900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145939456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.698905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.698908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.698909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697790.698932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697790.698942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.698948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f518738bb80 x1874531145939584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.698957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145939584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.698965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.698968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.698971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697790.698977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697790.698996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.699006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f518738a300 x1874531145939712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.699018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145939712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.699023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.699025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.699026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697790.699032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697790.699048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.699059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f5187389500 x1874531145939840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.699070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145939840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.699077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.699080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.699082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145939840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697790.699086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145939968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697790.699100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.699106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f518738ad80 x1874531145939968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.699113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145939968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.699123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.699125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.699135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145939968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697790.699143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145940096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697790.699154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.699160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f518738aa00 x1874531145940096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.699167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145940096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.699172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.699173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.699175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145940096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697790.699178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145940224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697790.699187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.699192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f5187389180 x1874531145940224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.699198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145940224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.699202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.699204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.699206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145940224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697790.699210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145940352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697790.699219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697790.699224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697790/real 1787697790] req@ffff8f5187388380 x1874531145940352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697790.699230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145940352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697790.699235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697790.699236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697790.699238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145940352:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697791.594601:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697792.618628:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697793.642533:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697794.666607:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697795.690518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697795.818424:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818435:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697795.818439:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697795.818449:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818460:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697795.818467:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697795.818502:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145940480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697795.818532:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818535:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697795.818537:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697795.818540:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818543:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697795.818546:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697795.818555:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697795.818567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145940480:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697795.818571:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145940608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697795.818583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697795.818588:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697795.818592:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697795.818601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697795.818606:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697795.818620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145940736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697795.818622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697795.818633:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818646:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697795.818648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697795.818653:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818657:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697795.818660:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697795.818665:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697795.818671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f518738b100 x1874531145940480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697795.818679:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145940864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697795.818692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818695:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697795.818697:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697795.818701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145940480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697795.818702:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818707:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697795.818718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697795.818719:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787697795.818726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697795.818727:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697795.818733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145940480:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697795.818739:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145940992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697795.818745:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818747:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697795.818748:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697795.818751:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697795.818755:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697795.818759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940608:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697795.818759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697795.818768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145941120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697795.818781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697795.818783:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818788:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697795.818790:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697795.818794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f518738b800 x1874531145940608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697795.818798:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818806:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697795.818810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145940608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697795.818812:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697795.818822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787697795.818823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.818828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.818831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697795.818839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145941248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697795.818845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697795.818852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818863:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697795.818866:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697795.818867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697795.818875:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00080000:3.0:1787697795.818880:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697795.818885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f518738a680 x1874531145940736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697795.818886:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697795.818893:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697795.818903:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145940736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697795.818909:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145941376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697795.818924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697795.818926:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697795.818931:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697795.818934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697795.818934:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697795.818940:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5695 00000100:00100000:1.0:1787697795.818944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697795.818945:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697795.818951:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697795.818955:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697795.818957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697795.818964:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145941504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697795.818984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697795.818997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f5187388000 x1874531145940864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697795.819015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145940864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697795.819023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.819025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.819035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697795.819039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697795.819057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697795.819067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f518738b480 x1874531145940992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697795.819077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145940992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697795.819084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.819087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.819090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145940992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697795.819106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697795.819122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697795.819132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f5187388e00 x1874531145941120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697795.819143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145941120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697795.819156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.819158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.819161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697795.819167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697795.819210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697795.819220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f5187389f80 x1874531145941248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697795.819227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145941248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697795.819232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.819234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.819236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697795.819239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697795.819251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697795.819257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f518738b100 x1874531145941376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697795.819263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145941376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697795.819267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.819268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.819271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697795.819274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697795.819282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697795.819287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697795/real 1787697795] req@ffff8f5187388380 x1874531145941504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697795.819293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145941504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697795.819297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697795.819298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697795.819300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145941504:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697796.714570:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697797.738541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697798.762468:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697799.786488:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697800.810638:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697800.938457:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697800.938471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938481:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938491:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938497:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697800.938536:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145941632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938562:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938566:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697800.938568:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938572:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938576:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938579:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938589:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697800.938601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145941760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697800.938611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145941632:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697800.938611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697800.938615:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938622:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938624:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938629:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697800.938636:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145941888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938642:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938644:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697800.938645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938647:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938649:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938650:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938653:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697800.938661:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145942016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938668:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938671:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697800.938673:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697800.938680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697800.938680:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938687:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938691:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938697:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697800.938707:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145942144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697800.938717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938719:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938721:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938724:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697800.938742:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145942272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697800.938743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704f800 x1874531145941632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697800.938753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697800.938761:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938767:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938769:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938772:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938775:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697800.938781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145941632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697800.938784:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145942400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938813:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938815:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697800.938816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697800.938816:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938823:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938825:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697800.938826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697800.938830:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938834:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697800.938835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145941632:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697800.938842:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145942528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697800.938848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697800.938849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697800.938850:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697800.938852:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5700 00000100:00080000:2.0:1787697800.938855:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697800.938856:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697800.938859:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697800.938865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145941760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697800.938870:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145942656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697800.938899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.938914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704d880 x1874531145941760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.938926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145941760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.938934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.938937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.938940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145941760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697800.938946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145941888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697800.938963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.938974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704c700 x1874531145941888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.938985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145941888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.938992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.938995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.938998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145941888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697800.939003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145942016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697800.939022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.939033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704c380 x1874531145942016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.939062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145942016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.939070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.939073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.939076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145942016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697800.939082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145942144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697800.939100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.939143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704c000 x1874531145942144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.939158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145942144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.939166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.939169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.939172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145942144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697800.939177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697800.939210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.939222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704e680 x1874531145942272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.939233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145942272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.939247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.939250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.939253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697800.939259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697800.939278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.939290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704ce00 x1874531145942400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.939302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145942400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.939309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.939312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.939315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697800.939321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697800.939339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.939349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704df80 x1874531145942528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.939391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145942528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.939399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.939402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.939405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697800.939427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697800.939446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697800.939458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff8f518704f800 x1874531145942656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.939469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145942656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.939477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.939480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697800.939483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145942656:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697801.834532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697802.858504:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697803.882532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697804.906504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697805.930575:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697806.058488:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058495:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697806.058498:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058507:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058517:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697806.058524:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697806.058561:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145942784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697806.058587:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058591:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697806.058593:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058596:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058598:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697806.058601:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058608:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697806.058616:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531145942912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697806.058622:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058625:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697806.058626:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697806.058630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145942784:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697806.058631:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058636:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697806.058640:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058643:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697806.058651:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531145943040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697806.058657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058659:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697806.058660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058662:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697806.058678:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697806.058684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697806.058694:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531145943168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697806.058703:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697806.058708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058713:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058717:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697806.058720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace90e00 x1874531145942784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697806.058724:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697806.058740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145942784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697806.058747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531145943296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697806.058749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.058756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787697806.058761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697806.058762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145942784:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787697806.058764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697806.058769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058773:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058776:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697806.058778:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697806.058786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145942912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697806.058789:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531145943424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697806.058802:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058804:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697806.058806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058808:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058810:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697806.058813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787697806.058817:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058820:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697806.058828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace92d80 x1874531145942912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697806.058830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531145943552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697806.058862:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058865:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697806.058867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697806.058870:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058875:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697806.058877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531145942912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697806.058882:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058888:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697806.058893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697806.058899:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531145943680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697806.058901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.058909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145942912:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697806.058911:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697806.058917:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697806.058918:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697806.058922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145943040:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787697806.058924:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5705 00000100:00080000:0.0:1787697806.058929:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697806.058931:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787697806.058934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697806.058942:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531145943808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697806.058953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.058963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace93480 x1874531145943040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.058971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531145943040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.058975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.058977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.058983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145943040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697806.058987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145943168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697806.058999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.059005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace92300 x1874531145943168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.059015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531145943168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.059020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.059022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.059044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145943168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697806.059049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145943296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697806.059061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.059068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace91500 x1874531145943296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.059074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531145943296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.059079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.059081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.059083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145943296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697806.059093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697806.059102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.059107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace91880 x1874531145943424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.059114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531145943424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.059131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.059133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.059135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697806.059138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787697806.059148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.059154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace90e00 x1874531145943552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.059160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531145943552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.059164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.059166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.059168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697806.059170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697806.059178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.059183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace93100 x1874531145943680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.059192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531145943680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.059198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.059200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.059204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697806.059209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787697806.059238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697806.059246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697806/real 1787697806] req@ffff8f51ace91c00 x1874531145943808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697806.059254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531145943808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697806.059259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697806.059260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697806.059262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145943808:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697806.954609:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697807.978512:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697809.002504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697810.026545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697811.050475:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697811.178447:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178454:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697811.178457:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178464:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178473:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178479:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697811.178507:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145943936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697811.178528:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178531:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697811.178532:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178535:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178538:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178540:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178547:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697811.178555:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145944064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697811.178563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145943936:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697811.178564:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178576:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697811.178577:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178580:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178583:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178585:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178588:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697811.178596:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145944192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697811.178602:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178604:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697811.178605:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178607:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178609:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178611:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178614:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697811.178622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697811.178622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145944320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697811.178635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697811.178640:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178644:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178654:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178658:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178663:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697811.178672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183647800 x1874531145943936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697811.178678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145944448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697811.178688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697811.178691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145943936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697811.178692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697811.178694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178701:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697811.178706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697811.178710:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697811.178713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697811.178716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697811.178718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145943936:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697811.178727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145944576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697811.178733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697811.178737:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697811.178740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944064:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697811.178741:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:1.0:1787697811.178746:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178748:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697811.178760:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145944704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697811.178767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697811.178773:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178776:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697811.178777:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178781:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00100000:0.0:1787697811.178782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183647100 x1874531145944064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697811.178783:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697811.178786:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178789:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697811.178799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145944832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697811.178803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145944064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697811.178810:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697811.178817:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697811.178818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697811.178818:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697811.178826:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5710 00000100:00080000:0.0:1787697811.178827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697811.178829:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697811.178832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944064:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697811.178834:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697811.178841:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697811.178844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697811.178849:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145944960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697811.178859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.178870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183645180 x1874531145944192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.178878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145944192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.178883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.178885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.178887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697811.178890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697811.178900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.178906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183645c00 x1874531145944320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.178912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145944320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.178916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.178918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.178919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697811.178922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697811.178930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.178936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183644e00 x1874531145944448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.178960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145944448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.178968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.178970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.178974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145944448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697811.178980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697811.178996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.179004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183647480 x1874531145944576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.179015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145944576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.179020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.179021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.179023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697811.179034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697811.179059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.179068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183646680 x1874531145944704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.179080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145944704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.179086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.179088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.179090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697811.179094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697811.179109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.179119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183644000 x1874531145944832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.179129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145944832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.179142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.179145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.179147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697811.179150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697811.179163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697811.179169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697811/real 1787697811] req@ffff8f5183644700 x1874531145944960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697811.179178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145944960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697811.179185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697811.179188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697811.179190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145944960:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697812.074495:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697813.098494:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697814.122517:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697815.146486:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697816.170552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697816.298466:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298473:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697816.298475:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298483:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298491:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298496:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697816.298529:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145945088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298549:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697816.298552:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298556:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298560:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298567:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697816.298575:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145945216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697816.298587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697816.298594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145945088:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697816.298597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298604:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298608:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298613:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697816.298627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145945344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298635:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298637:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697816.298639:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298641:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298644:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298646:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298649:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697816.298657:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145945472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298663:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697816.298670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298672:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298675:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298679:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697816.298690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697816.298697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145945600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298707:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298711:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697816.298713:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298718:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298726:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697816.298733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697816.298748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145945728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697816.298761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945472:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697816.298768:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697816.298771:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298776:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298781:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697816.298782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945600:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697816.298787:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298793:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697816.298800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145945728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697816.298807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145945856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298817:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298820:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697816.298823:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697816.298828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145945856:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697816.298828:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298836:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298841:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298846:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697816.298872:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145945984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697816.298884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697816.298888:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697816.298890:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697816.298897:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5715 00000100:00080000:3.0:1787697816.298901:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697816.298904:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697816.298908:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697816.298909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145945984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697816.298921:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145946112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697816.298968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145946112:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697817.194501:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697818.218539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697819.242512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697819.370632:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787697819.370672:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697819.370689:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370741:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370752:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370760:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370766:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370773:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370780:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370786:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697819.370793:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787697819.370887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f5187388380 x1874531145946112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.370911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145946112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.370922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.370927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.370932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145946112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.370940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f5187388a80 x1874531145945984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.370951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145945984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.370959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.370962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.370966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145945984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.370973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f5187389c00 x1874531145945856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.370991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145945856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.370998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145945856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697819.371010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f518738bb80 x1874531145945728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.371020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145945728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.371029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145945728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697819.371052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f518738ad80 x1874531145945344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.371064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145945344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.371071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697819.371083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f5187389500 x1874531145945472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.371096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145945472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.371103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.371114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f518738a300 x1874531145945600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.371125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145945600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.371132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697819.371144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f5187389180 x1874531145945088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.371155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145945088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.371161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145945088:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697819.371171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697816/real 1787697819] req@ffff8f518738aa00 x1874531145945216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697819.371182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145945216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697819.371189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697819.371192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697819.371194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145945216:192.168.201.125@tcp:38:kworker.0 00010000:00010000:1.0:1787697820.266532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697821.290527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697821.418501:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418509:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697821.418512:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418522:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418532:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418539:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697821.418569:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145946240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418598:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418609:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697821.418610:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418618:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418622:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418625:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697821.418635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697821.418647:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145946368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418655:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697821.418660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418663:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418666:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418669:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697821.418673:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697821.418683:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145946496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418695:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697821.418696:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418700:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418704:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418709:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697821.418711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145946240:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697821.418714:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697821.418727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145946624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418736:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418739:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697821.418741:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418748:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418752:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697821.418756:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697821.418784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145946752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418793:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418797:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697821.418800:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418805:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418808:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418811:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697821.418816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697821.418819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697821.418831:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145946880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418841:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697821.418847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418851:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418855:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697821.418860:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697821.418865:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697821.418868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183646d80 x1874531145946240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697821.418880:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145947008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697821.418889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145946240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697821.418893:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418901:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697821.418906:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697821.418910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697821.418911:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418919:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697821.418920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697821.418925:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697821.418931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145946240:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697821.418934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697821.418945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697821.418950:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145947136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697821.418960:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697821.418964:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697821.418966:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697821.418969:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5721 00000100:00080000:1.0:1787697821.418973:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697821.418977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697821.418982:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697821.418988:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697821.418996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183646a00 x1874531145946368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697821.419001:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145947264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697821.419023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145946368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697821.419052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697821.419070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183647b80 x1874531145946496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145946496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697821.419145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697821.419166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183644380 x1874531145946624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145946624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697821.419209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697821.419226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183644700 x1874531145946752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145946752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145946752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697821.419266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145946880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697821.419283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183644000 x1874531145946880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145946880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145946880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697821.419323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145947008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697821.419341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183646680 x1874531145947008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145947008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145947008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697821.419434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145947136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697821.419453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183647480 x1874531145947136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145947136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145947136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697821.419492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145947264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697821.419508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697821.419517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697821/real 1787697821] req@ffff8f5183644e00 x1874531145947264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697821.419527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145947264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697821.419536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697821.419538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697821.419542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145947264:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697822.314520:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697823.338541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697824.362521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697825.386498:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697826.410494:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697826.538453:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538460:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697826.538463:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538471:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538479:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538484:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697826.538511:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145947392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538530:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538533:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697826.538534:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538537:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538539:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538541:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538547:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697826.538555:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145947520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538561:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538564:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697826.538565:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538567:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538571:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538574:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697826.538579:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145947648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697826.538584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145947392:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697826.538586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697826.538593:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538597:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538599:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538601:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697826.538611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145947776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697826.538619:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538621:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538623:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538624:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697826.538634:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145947904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697826.538642:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538644:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538647:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538651:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697826.538652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697826.538662:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145948032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697826.538672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538687:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538691:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538695:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538698:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697826.538710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183645c00 x1874531145947392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697826.538710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145948160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538730:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697826.538731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697826.538737:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538740:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697826.538742:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538745:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697826.538749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145947392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697826.538753:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145948288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697826.538774:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697826.538777:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697826.538778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697826.538779:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697826.538787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697826.538787:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5726 00000100:00080000:1.0:1787697826.538793:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697826.538795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145947392:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697826.538800:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697826.538806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697826.538808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697826.538821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145948416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697826.538835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.538849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183645180 x1874531145947520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.538861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145947520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.538869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.538872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.538876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697826.538898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697826.538915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.538924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183647100 x1874531145947648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.538935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145947648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.538943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.538945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.538949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697826.538954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697826.538969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.538978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183647800 x1874531145947776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.538989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145947776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.538996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.538999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.539002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697826.539007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697826.539022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.539032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183644e00 x1874531145947904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.539042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145947904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.539050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.539053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.539056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145947904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697826.539061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697826.539076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.539086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183647480 x1874531145948032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.539097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145948032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.539104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.539107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.539110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697826.539115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697826.539132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.539141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183646680 x1874531145948160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.539151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145948160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.539160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.539163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.539166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697826.539171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697826.539185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.539194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183644000 x1874531145948288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.539204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145948288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.539211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.539214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.539217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697826.539222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697826.539236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697826.539253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697826/real 1787697826] req@ffff8f5183644700 x1874531145948416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697826.539265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145948416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697826.539272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697826.539274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697826.539278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145948416:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697827.434528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697828.458537:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697829.482478:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697830.506508:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697831.530517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697831.658486:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658494:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697831.658498:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658508:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658518:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658524:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697831.658558:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145948544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697831.658577:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658579:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697831.658581:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658584:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658607:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697831.658619:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145948672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697831.658624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145948544:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697831.658628:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658631:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697831.658633:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658635:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658638:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658640:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658642:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697831.658649:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145948800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697831.658658:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697831.658662:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658664:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658666:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658668:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658672:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697831.658681:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145948928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697831.658689:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787697831.658695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697831.658695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658702:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658706:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658711:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697831.658728:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145949056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697831.658734:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658738:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697831.658739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704fb80 x1874531145948544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697831.658740:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658746:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658751:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658755:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658759:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697831.658765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145948544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697831.658768:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145949184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697831.658789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697831.658790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697831.658791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697831.658794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697831.658798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697831.658800:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658804:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697831.658807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145948544:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697831.658809:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658814:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697831.658821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145949312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697831.658826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658828:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697831.658830:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658833:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00100000:0.0:1787697831.658834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145948672:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697831.658835:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697831.658842:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658847:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697831.658859:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145949440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697831.658863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.658875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704dc00 x1874531145948672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697831.658881:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697831.658887:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697831.658889:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697831.658894:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5731 00000100:00080000:2.0:1787697831.658896:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697831.658898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145948672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697831.658901:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697831.658908:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697831.658910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.658915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.658918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145948672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697831.658921:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145949568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697831.658925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145948800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697831.658944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.658951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704d180 x1874531145948800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.658966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145948800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.658971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.658973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.658975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145948800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697831.658979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145948928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697831.658989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.658995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704d500 x1874531145948928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.659001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145948928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.659010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.659012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.659014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145948928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697831.659017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145949056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697831.659027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.659032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704f480 x1874531145949056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.659042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145949056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.659047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.659048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.659050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145949056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697831.659053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697831.659061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.659066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704e300 x1874531145949184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.659073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145949184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.659078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.659079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.659080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697831.659083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697831.659091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.659096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704ea00 x1874531145949312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.659102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145949312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.659110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.659112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.659114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697831.659125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697831.659133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.659138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704ed80 x1874531145949440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.659144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145949440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.659148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.659150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.659151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697831.659154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697831.659163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697831.659167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697831/real 1787697831] req@ffff8f518704fb80 x1874531145949568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697831.659173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145949568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697831.659177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697831.659179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697831.659181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145949568:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697832.554545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697833.578541:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697834.602560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697835.626541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697836.650536:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697836.778495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778517:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697836.778521:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778531:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778541:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778548:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697836.778597:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145949696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778642:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697836.778643:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778648:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778652:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778656:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697836.778680:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145949824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778689:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697836.778693:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778695:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778698:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778701:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778705:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697836.778715:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145949952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778724:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778727:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697836.778729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778732:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778735:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778747:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697836.778764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145950080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778773:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778776:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697836.778778:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778782:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778784:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778788:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697836.778800:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145950208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697836.778813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778816:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778819:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778822:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778826:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697836.778835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145950336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697836.778856:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778862:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778865:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778868:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778873:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697836.778883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145950464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778892:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778894:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697836.778896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778899:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778902:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778905:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778909:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697836.778917:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145950592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697836.778925:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697836.778928:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697836.778929:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697836.778933:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5736 00000100:00080000:2.0:1787697836.778935:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697836.778938:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697836.778943:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697836.778951:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145950720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697836.779070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145949696:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787697836.779128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704f800 x1874531145949696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145949696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145949696:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697836.779213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145949824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697836.779234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704df80 x1874531145949824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145949824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145949824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697836.779275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145949952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697836.779293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704ce00 x1874531145949952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145949952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145949952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697836.779333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145950080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697836.779350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704e680 x1874531145950080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145950080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145950080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697836.779481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145950208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697836.779514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704c000 x1874531145950208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145950208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145950208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697836.779563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697836.779582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704c380 x1874531145950336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145950336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697836.779622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697836.779640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704c700 x1874531145950464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145950464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697836.779686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697836.779704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704d880 x1874531145950592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779725:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145950592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697836.779745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697836.779762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697836.779772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697836/real 1787697836] req@ffff8f518704fb80 x1874531145950720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697836.779783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145950720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697836.779792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697836.779794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697836.779798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145950720:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697837.674540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697838.698509:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697839.722517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697840.746537:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697841.770533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697841.898455:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898464:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697841.898467:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898477:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898486:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.898493:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697841.898528:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145950848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697841.898554:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898558:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697841.898560:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898564:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898568:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.898573:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697841.898581:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697841.898596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145950848:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697841.898596:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145950976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697841.898613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697841.898618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898621:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898625:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.898628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697841.898633:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697841.898646:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145951104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697841.898656:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898659:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787697841.898663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697841.898663:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898675:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.898680:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697841.898685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697841.898696:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145951232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697841.898704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f518738b100 x1874531145950848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697841.898706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898717:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697841.898722:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898727:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00100000:0.0:1787697841.898730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145950848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697841.898732:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.898741:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697841.898743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697841.898746:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697841.898754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.898757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145950848:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697841.898762:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145951360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697841.898772:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898776:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697841.898779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145950976:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697841.898780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898788:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898792:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.898796:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697841.898801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697841.898802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697841.898812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f5187389f80 x1874531145950976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697841.898818:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145951488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697841.898832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145950976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697841.898835:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697841.898844:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697841.898848:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697841.898852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697841.898855:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:0.0:1787697841.898859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697841.898861:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697841.898867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145950976:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697841.898870:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697841.898878:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697841.898879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145951104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697841.898894:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145951616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697841.898905:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697841.898909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697841.898911:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697841.898915:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898920:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898924:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697841.898926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f5187388e00 x1874531145951104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697841.898932:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697841.898941:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697841.898948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145951104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697841.898956:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145951744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697841.898958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.898965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.898973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145951104:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697841.898976:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697841.898981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145951232:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697841.898981:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697841.898985:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697841.898992:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5741 00000100:00080000:3.0:1787697841.898997:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697841.899001:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697841.899004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697841.899010:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697841.899015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f518738b480 x1874531145951232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697841.899029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145951232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697841.899030:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145951872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697841.899046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.899048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.899050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145951232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697841.899054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145951360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697841.899070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697841.899077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f5187388000 x1874531145951360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697841.899088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145951360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697841.899092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.899094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.899096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145951360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697841.899106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697841.899116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697841.899122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f518738a680 x1874531145951488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697841.899133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145951488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697841.899140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.899144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.899147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697841.899152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697841.899168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697841.899176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f518738b800 x1874531145951616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697841.899183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145951616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697841.899191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.899193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.899195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697841.899198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697841.899207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697841.899213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f518738b100 x1874531145951744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697841.899219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145951744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697841.899223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.899224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.899226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697841.899228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697841.899236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697841.899241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff8f518738aa00 x1874531145951872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697841.899247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145951872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697841.899251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697841.899252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697841.899254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145951872:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697842.794557:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697843.818505:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697844.842558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697845.866474:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697846.890535:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697847.018415:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018422:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697847.018425:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697847.018433:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018441:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697847.018446:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697847.018476:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145952000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697847.018495:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018507:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697847.018509:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697847.018515:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018519:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697847.018524:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697847.018532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145952000:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697847.018534:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697847.018551:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145952128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697847.018561:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018565:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697847.018566:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697847.018570:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018573:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697847.018577:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697847.018580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697847.018584:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697847.018598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145952256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697847.018607:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697847.018611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697847.018614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f5187389180 x1874531145952000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697847.018617:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018623:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697847.018628:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697847.018632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145952000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697847.018633:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697847.018644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697847.018648:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145952384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697847.018649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.018654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145952000:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697847.018668:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018671:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697847.018672:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697847.018675:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018678:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697847.018681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952128:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697847.018684:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697847.018692:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697847.018708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697847.018714:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145952512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697847.018723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f518738a300 x1874531145952128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697847.018727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018735:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697847.018739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697847.018742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145952128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697847.018746:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697847.018756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697847.018759:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697847.018763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787697847.018766:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697847.018781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697847.018781:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145952640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697847.018792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952256:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697847.018796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018803:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697847.018805:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697847.018809:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018813:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697847.018816:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697847.018821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697847.018822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697847.018834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f5187389500 x1874531145952256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697847.018845:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145952768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697847.018848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145952256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697847.018867:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697847.018869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697847.018874:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697847.018877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697847.018878:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697847.018884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952256:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697847.018885:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018890:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697847.018896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952384:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697847.018897:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697847.018904:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697847.018917:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145952896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697847.018927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697847.018927:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697847.018933:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697847.018935:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697847.018939:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5746 00000100:00080000:3.0:1787697847.018942:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697847.018943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f518738ad80 x1874531145952384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697847.018948:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697847.018957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697847.018963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145952384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697847.018971:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145953024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697847.018973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.018989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.018993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697847.019008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697847.019024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697847.019032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f518738bb80 x1874531145952512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.019049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145952512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.019057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.019060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.019062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145952512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697847.019067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145952640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697847.019079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697847.019088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f5187389c00 x1874531145952640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.019098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145952640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.019106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.019109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.019112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145952640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697847.019117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145952768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697847.019146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697847.019155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f5187388a80 x1874531145952768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.019166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145952768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.019179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.019182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.019185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145952768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697847.019190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145952896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697847.019205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697847.019215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f5187388380 x1874531145952896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.019226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145952896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.019234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.019237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.019240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145952896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697847.019245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145953024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697847.019261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697847.019270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697847/real 1787697847] req@ffff8f5187389180 x1874531145953024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697847.019280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145953024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697847.019287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697847.019290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697847.019293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145953024:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697847.914528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697848.938534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697849.962535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697850.986569:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697852.010558:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697852.138477:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138486:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697852.138490:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138500:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138511:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138518:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697852.138561:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145953152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697852.138585:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138588:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697852.138589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138595:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138605:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697852.138615:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145953280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697852.138622:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138624:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697852.138625:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138628:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138629:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138634:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697852.138640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145953408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697852.138648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145953152:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697852.138648:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138651:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697852.138655:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138661:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138665:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138668:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138673:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697852.138685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145953536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697852.138694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138698:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697852.138699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138703:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138724:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138729:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697852.138734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697852.138751:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145953664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697852.138759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697852.138763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138767:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138770:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138773:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697852.138782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183644380 x1874531145953152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697852.138787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145953792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697852.138818:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138821:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697852.138822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138825:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00100000:0.0:1787697852.138829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145953152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697852.138829:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138835:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138840:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697852.138849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697852.138850:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145953920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697852.138858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697852.138861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697852.138865:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145953152:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697852.138866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697852.138868:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138876:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5751 00000100:00080000:1.0:1787697852.138878:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697852.138880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697852.138883:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138888:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697852.138896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145954048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697852.138901:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697852.138903:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697852.138905:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697852.138909:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5751 00000800:00000100:0.0:1787697852.138914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697852.138914:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697852.138920:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697852.138924:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697852.138930:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145954176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697852.138945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183647b80 x1874531145953280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.138958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145953280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.138964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.138967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.138969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697852.138997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697852.139014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183646a00 x1874531145953408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145953408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697852.139055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697852.139075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183646d80 x1874531145953536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145953536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697852.139107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697852.139122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183644700 x1874531145953664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145953664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145953664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697852.139168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145953792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697852.139182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183644000 x1874531145953792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145953792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145953792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697852.139219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145953920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697852.139251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183646680 x1874531145953920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145953920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145953920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697852.139285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145954048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697852.139298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183647480 x1874531145954048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145954048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145954048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697852.139320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145954176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697852.139330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697852.139335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697852/real 1787697852] req@ffff8f5183644e00 x1874531145954176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697852.139341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145954176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697852.139346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697852.139347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697852.139350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145954176:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697853.034570:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697854.058563:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697855.082530:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697856.106485:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697857.130509:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697857.258423:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258430:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697857.258433:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258441:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258449:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258454:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697857.258480:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145954304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697857.258497:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258500:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697857.258502:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258505:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258507:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258510:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258517:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697857.258525:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145954432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697857.258534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145954304:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697857.258534:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258539:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697857.258541:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258546:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258547:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258549:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258552:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697857.258561:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145954560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697857.258566:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258569:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697857.258571:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258575:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258578:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258582:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258587:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697857.258592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697857.258601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145954688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697857.258611:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697857.258616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258620:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258623:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258625:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258629:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697857.258632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f518738aa00 x1874531145954304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697857.258640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145954816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697857.258650:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697857.258654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697857.258656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145954304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697857.258659:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258663:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258668:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697857.258678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697857.258683:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145954944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697857.258686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.258690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145954304:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697857.258704:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258706:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697857.258708:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258712:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258714:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258716:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697857.258720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697857.258728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145955072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697857.258739:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697857.258741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697857.258743:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258747:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5756 00000800:00000100:0.0:1787697857.258748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697857.258750:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258754:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697857.258758:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697857.258766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f518738b100 x1874531145954432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697857.258767:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145955200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697857.258789:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697857.258792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145954432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697857.258793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697857.258796:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697857.258802:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5756 00000100:00080000:3.0:1787697857.258807:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697857.258812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697857.258813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787697857.258815:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697857.258820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697857.258825:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145955328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697857.258826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697857.258839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697857.258862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.258871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f518738b800 x1874531145954560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.258882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145954560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.258889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.258891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.258895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697857.258900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697857.258915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.258924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f518738a680 x1874531145954688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.258933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145954688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.258940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.258943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.258946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697857.258951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697857.258964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.258972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f5187388000 x1874531145954816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.258989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145954816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.258996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.258999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.259001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145954816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697857.259006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145954944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697857.259020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.259029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f518738b480 x1874531145954944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.259039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145954944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.259047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.259050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.259052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145954944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697857.259068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145955072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697857.259083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.259092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f5187388e00 x1874531145955072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.259102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145955072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.259114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.259118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.259133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145955072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697857.259138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145955200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697857.259171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.259181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f5187389f80 x1874531145955200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.259191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145955200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.259198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.259201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.259204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145955200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697857.259209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145955328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697857.259223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697857.259232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697857/real 1787697857] req@ffff8f518738aa00 x1874531145955328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697857.259241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145955328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697857.259247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697857.259250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697857.259254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145955328:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697858.154475:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697859.178534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697860.202500:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697861.226548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697862.250466:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697862.378461:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697862.378473:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697862.378477:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378487:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378499:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697862.378515:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697862.378552:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145955456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697862.378578:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697862.378582:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697862.378583:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378588:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378592:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697862.378596:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697862.378605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697862.378622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145955456:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787697862.378622:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145955584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697862.378638:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697862.378640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697862.378642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378647:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378651:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697862.378654:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697862.378659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697862.378672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145955712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697862.378678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697862.378687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697862.378691:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697862.378693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378698:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378702:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697862.378706:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697862.378711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f5187389180 x1874531145955456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697862.378712:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697862.378730:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145955840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697862.378731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145955456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697862.378760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697862.378761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697862.378765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697862.378767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697862.378770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145955456:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697862.378772:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378779:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378783:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697862.378787:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697862.378792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955584:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697862.378792:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697862.378811:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145955968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697862.378815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697862.378826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f5187388380 x1874531145955584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697862.378829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697862.378837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697862.378839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378844:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378850:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697862.378855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145955584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697862.378856:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697862.378865:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697862.378866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697862.378871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.378874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697862.378881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787697862.378882:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145956096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697862.378896:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697862.378899:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787697862.378903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697862.378903:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.378910:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378914:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697862.378917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f5187388a80 x1874531145955712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697862.378921:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697862.378928:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697862.378937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145955712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697862.378943:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145956224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697862.378955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697862.378960:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697862.378963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697862.378965:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697862.378969:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697862.378971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955712:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697862.378975:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.378981:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697862.378984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955840:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787697862.378988:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697862.378996:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697862.379012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697862.379026:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145956352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697862.379027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f5187389c00 x1874531145955840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697862.379051:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697862.379055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145955840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697862.379056:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697862.379060:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697862.379078:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5762 00000100:00080000:3.0:1787697862.379081:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697862.379085:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697862.379092:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697862.379101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697862.379106:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145956480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697862.379112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.379125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697862.379146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697862.379166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697862.379177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f518738bb80 x1874531145955968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697862.379194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145955968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697862.379201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697862.379204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.379207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145955968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697862.379213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697862.379229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697862.379238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f518738ad80 x1874531145956096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697862.379248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145956096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697862.379255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697862.379258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.379261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697862.379266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697862.379282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697862.379292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f5187389500 x1874531145956224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697862.379302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145956224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697862.379316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697862.379319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.379322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697862.379327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697862.379341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697862.379350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f518738a300 x1874531145956352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697862.379391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145956352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697862.379399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697862.379403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.379406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697862.379411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697862.379428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697862.379452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697862/real 1787697862] req@ffff8f5187389180 x1874531145956480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697862.379463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145956480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697862.379471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697862.379474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697862.379478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145956480:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697863.274494:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697864.298584:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697865.322586:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697866.346835:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697867.370584:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697867.498456:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498465:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697867.498469:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498481:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:1.0:1787697867.498493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697867.498499:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697867.498541:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145956608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697867.498575:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697867.498586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498592:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:1.0:1787697867.498595:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697867.498599:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697867.498609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697867.498621:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145956736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697867.498623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145956608:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697867.498637:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498640:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697867.498642:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498646:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:1.0:1787697867.498649:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697867.498653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697867.498658:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697867.498671:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145956864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697867.498681:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498685:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697867.498687:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498691:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:1.0:1787697867.498698:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697867.498700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697867.498704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697867.498710:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697867.498720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145956992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697867.498726:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697867.498730:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498733:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:1.0:1787697867.498736:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697867.498739:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697867.498742:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697867.498746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183647800 x1874531145956608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697867.498754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145957120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697867.498761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697867.498762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145956608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697867.498766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697867.498769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498777:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:0.0:1787697867.498779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697867.498780:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697867.498787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697867.498788:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697867.498793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145956608:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787697867.498794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697867.498807:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145957248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697867.498813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498815:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697867.498816:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697867.498819:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00100000:0.0:1787697867.498821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145956736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697867.498823:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697867.498829:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697867.498833:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697867.498844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697867.498847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145957376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697867.498854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183647100 x1874531145956736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.498876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145956736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697867.498877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498886:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697867.498890:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697867.498891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.498896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697867.498899:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00100000:0.0:1787697867.498901:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145956736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697867.498908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145956864:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697867.498908:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697867.498917:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697867.498922:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697867.498931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697867.498937:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145957504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697867.498940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183645180 x1874531145956864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697867.498958:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697867.498961:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697867.498963:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697867.498967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145956864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697867.498968:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5767 00000100:00080000:1.0:1787697867.498988:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697867.498991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697867.498994:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697867.498999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697867.498999:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697867.499007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145956864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697867.499012:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145957632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697867.499016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145956992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697867.499041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697867.499050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183645c00 x1874531145956992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.499057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145956992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697867.499062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.499064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697867.499066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145956992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697867.499069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145957120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697867.499081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697867.499090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183644e00 x1874531145957120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.499122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145957120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697867.499128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.499130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697867.499132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145957120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697867.499136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697867.499151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697867.499158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183647480 x1874531145957248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.499165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145957248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697867.499170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.499172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697867.499174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697867.499186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697867.499196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697867.499201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183646680 x1874531145957376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.499207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145957376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697867.499216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.499218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697867.499219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697867.499222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697867.499230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697867.499235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183644000 x1874531145957504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.499241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145957504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697867.499246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.499248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697867.499250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697867.499252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697867.499260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697867.499265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697867/real 1787697867] req@ffff8f5183644700 x1874531145957632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697867.499270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145957632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697867.499275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697867.499277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697867.499279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145957632:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697868.394526:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697869.418511:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697870.442538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697871.466544:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697872.490519:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697872.618494:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697872.618504:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697872.618509:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618519:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:3.0:1787697872.618530:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697872.618538:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697872.618574:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145957760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697872.618602:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697872.618606:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697872.618608:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618613:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:3.0:1787697872.618616:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697872.618619:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.618628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697872.618641:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145957888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697872.618650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145957760:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697872.618654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697872.618666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697872.618667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618672:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:3.0:1787697872.618676:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697872.618679:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.618684:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697872.618697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145958016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697872.618707:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697872.618710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697872.618712:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697872.618715:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618720:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:3.0:1787697872.618725:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697872.618729:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.618735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697872.618746:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145958144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697872.618752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f518738aa00 x1874531145957760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697872.618758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697872.618766:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697872.618769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:3.0:1787697872.618778:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697872.618779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145957760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697872.618784:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.618793:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697872.618796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.618802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787697872.618807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145958272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697872.618808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145957760:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697872.618822:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697872.618825:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697872.618827:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618832:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00100000:0.0:1787697872.618835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145957888:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697872.618846:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697872.618855:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697872.618860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787697872.618867:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697872.618881:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145958400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697872.618885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f5187389f80 x1874531145957888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697872.618916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145957888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697872.618934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697872.618936:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697872.618939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697872.618942:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697872.618946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145957888:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697872.618947:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.618956:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00100000:0.0:1787697872.618958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145958016:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697872.618961:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697872.618968:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.618973:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697872.618985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697872.618989:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145958528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697872.618999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f5187388e00 x1874531145958016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697872.619023:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697872.619025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145958016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697872.619030:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697872.619034:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697872.619043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697872.619044:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:0.0:1787697872.619050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697872.619051:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697872.619056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145958016:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697872.619059:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.619066:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697872.619067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145958144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697872.619082:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145958656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697872.619092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697872.619096:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697872.619101:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697872.619105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f518738b480 x1874531145958144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697872.619106:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697872.619114:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5772 00000100:00080000:3.0:1787697872.619121:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697872.619126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145958144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697872.619128:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697872.619136:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697872.619138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.619144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697872.619151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145958144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697872.619152:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145958784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697872.619161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145958272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697872.619183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697872.619194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f5187388000 x1874531145958272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697872.619211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145958272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697872.619218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.619221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697872.619224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145958272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697872.619240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697872.619258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697872.619269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f518738a680 x1874531145958400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697872.619292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145958400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697872.619300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.619303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697872.619307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697872.619312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697872.619330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697872.619339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f518738b800 x1874531145958528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697872.619350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145958528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697872.619392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.619396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697872.619399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697872.619404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697872.619424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697872.619435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f518738b100 x1874531145958656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697872.619446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145958656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697872.619453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.619456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697872.619459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697872.619464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697872.619480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697872.619489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697872/real 1787697872] req@ffff8f518738aa00 x1874531145958784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697872.619498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145958784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697872.619506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697872.619508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697872.619512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145958784:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697873.514534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697874.538523:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697875.562625:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697876.586496:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697877.610495:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697877.738455:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.738464:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697877.738469:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697877.738479:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00080000:1.0:1787697877.738489:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697877.738505:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697877.738537:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145958912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697877.738564:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.738567:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697877.738569:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697877.738574:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00080000:1.0:1787697877.738577:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697877.738581:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697877.738590:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697877.738612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145958912:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697877.738612:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145959040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697877.738629:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.738632:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697877.738634:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697877.738639:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00080000:1.0:1787697877.738643:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697877.738647:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697877.738652:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697877.738663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145959168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697877.738666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697877.738683:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.738686:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697877.738688:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697877.738693:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00080000:1.0:1787697877.738697:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697877.738701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183646d80 x1874531145958912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.738704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697877.738712:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697877.738720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145958912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697877.738726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145959296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697877.738730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.738738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.738744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145958912:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697877.738754:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.738759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697877.738761:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697877.738765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959040:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697877.738767:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00080000:1.0:1787697877.738774:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697877.738778:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697877.738783:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697877.738808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697877.738812:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145959424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697877.738824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183646a00 x1874531145959040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.738826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.738835:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697877.738838:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697877.738842:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00100000:0.0:1787697877.738851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145959040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697877.738857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.738859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.738864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697877.738869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697877.738893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697877.738912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183647b80 x1874531145959168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.738919:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697877.738936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145959168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697877.738945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.738950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.738954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697877.738961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959296:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697877.738971:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697877.738982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697877.738996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183644380 x1874531145959296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697877.739009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145959296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697877.739014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.739016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697877.739018:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697877.739019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697877.739039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697877.739039:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145959552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697877.739055:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.739059:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787697877.739060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697877.739070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183646d80 x1874531145959424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.739071:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697877.739080:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00080000:1.0:1787697877.739087:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697877.739092:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697877.739097:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697877.739101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145959424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697877.739108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697877.739110:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145959680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697877.739113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.739119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145959424:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697877.739130:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697877.739133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697877.739136:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697877.739140:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697877.739150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697877.739154:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00100000:0.0:1787697877.739158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183644700 x1874531145959552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.739161:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697877.739168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145959552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.739170:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697877.739179:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697877.739180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.739186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.739189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697877.739196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697877.739198:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145959808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697877.739216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697877.739217:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697877.739222:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697877.739224:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697877.739227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183644000 x1874531145959680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.739231:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5777 00000100:00100000:0.0:1787697877.739241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145959680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697877.739246:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697877.739254:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697877.739259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697877.739262:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697877.739265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.739271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697877.739277:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145959936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697877.739297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697877.739318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697877.739325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183646680 x1874531145959808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697877.739332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145959808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697877.739337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.739338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.739340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697877.739343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697877.739352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697877.739413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697877/real 1787697877] req@ffff8f5183647480 x1874531145959936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697877.739421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145959936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697877.739432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697877.739435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697877.739438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145959936:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697878.634515:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697879.658540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697880.682530:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697881.706542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697882.730527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697882.858442:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858451:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697882.858455:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858465:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858474:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858481:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697882.858514:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145960064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858542:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858546:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697882.858547:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858551:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858555:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858559:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697882.858567:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697882.858580:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145960192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858590:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697882.858595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145960064:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697882.858595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697882.858599:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858606:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858609:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858612:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697882.858617:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697882.858627:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145960320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858636:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858639:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697882.858641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858645:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858649:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858653:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697882.858657:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697882.858668:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145960448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858687:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697882.858693:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858706:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697882.858707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960192:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697882.858723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697882.858738:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145960576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858748:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858752:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697882.858755:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697882.858756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960320:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697882.858763:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858768:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858772:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697882.858777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697882.858790:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145960704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697882.858796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960448:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697882.858801:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697882.858808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858811:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858822:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858827:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697882.858830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960576:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697882.858834:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697882.858847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145960832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858856:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697882.858859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697882.858862:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858866:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858869:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697882.858873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145960704:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697882.858874:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697882.858882:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697882.858895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145960960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697882.858905:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697882.858909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145960832:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697882.858910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697882.858913:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697882.858921:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5782 00000100:00080000:1.0:1787697882.858925:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697882.858928:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697882.858933:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697882.858946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145960960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697882.858947:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145961088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697882.858977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145961088:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697883.754556:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697884.778488:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697885.802550:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697885.930594:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787697885.930631:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697885.930646:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930704:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930717:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930724:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930731:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930738:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930745:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930752:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787697885.930759:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787697885.930832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183644700 x1874531145961088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.930853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145961088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.930876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.930881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.930885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145961088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.930893:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183644000 x1874531145960960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.930904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145960960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.930911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.930914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.930916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145960960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.930921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183646680 x1874531145960832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.930930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145960832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.930936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.930939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.930941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145960832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697885.930947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183647800 x1874531145960576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.930956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145960576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.930963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.930966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.930969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.930975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183647480 x1874531145960704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.930986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145960704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.930994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.930997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.931000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145960704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697885.931006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183645c00 x1874531145960192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.931017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145960192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.931024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.931027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.931030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697885.931036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183645180 x1874531145960320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.931046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145960320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.931053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.931056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.931058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697885.931070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183647100 x1874531145960448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.931080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145960448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.931086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.931089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.931092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145960448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697885.931097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697885] req@ffff8f5183644e00 x1874531145960064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697885.931107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145960064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697885.931114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697885.931116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697885.931119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145960064:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787697886.826514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697887.850549:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697887.978508:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978519:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697887.978522:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978532:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978542:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978549:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697887.978580:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145961216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697887.978606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978609:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697887.978611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978615:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978618:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978622:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697887.978648:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697887.978661:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145961344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697887.978672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145961216:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697887.978673:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697887.978683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978687:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978691:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978695:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697887.978699:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697887.978712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145961472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697887.978721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978724:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697887.978726:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978730:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978733:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978737:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697887.978742:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697887.978752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697887.978754:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145961600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697887.978764:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978768:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697887.978770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978777:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978781:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697887.978786:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697887.978794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f5187389180 x1874531145961216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697887.978800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145961728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697887.978817:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978821:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697887.978823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145961216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697887.978824:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978833:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978837:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978841:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697887.978846:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697887.978853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697887.978859:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145961856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697887.978862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.978869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145961216:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697887.978876:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.978880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697887.978882:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978887:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978890:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.978894:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697887.978898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961344:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697887.978901:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697887.978916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145961984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697887.978927:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697887.978928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697887.978933:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697887.978935:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.978940:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.978943:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697887.978946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f518738a300 x1874531145961344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697887.978950:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697887.978964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145961344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.978973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.978976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.978979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697887.978985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697887.979002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.979013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f5187389500 x1874531145961472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.979024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145961472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.979032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.979035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.979038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697887.979043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697887.979061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.979072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f518738ad80 x1874531145961600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.979082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145961600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.979089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.979091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.979095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697887.979099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697887.979122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.979132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f518738bb80 x1874531145961728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.979142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145961728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.979150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.979153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.979156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145961728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697887.979162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145961856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697887.979179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.979189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f5187389c00 x1874531145961856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.979207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145961856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.979216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.979218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.979220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145961856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697887.979245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145961984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697887.979264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.979274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f5187388a80 x1874531145961984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.979285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145961984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.979293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.979296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.979298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145961984:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697887.981922:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697887.981969:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145962112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697887.981996:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697887.982000:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697887.982002:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697887.982007:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5787 00000100:00080000:3.0:1787697887.982015:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697887.982021:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697887.982039:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697887.982046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145962112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787697887.982054:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145962240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697887.982105:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.982127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f5187388380 x1874531145962112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.982142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145962112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.982156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.982160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.982164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145962112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697887.982180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145962240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697887.982198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697887.982207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff8f5187388a80 x1874531145962240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697887.982218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145962240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697887.982225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697887.982228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697887.982231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145962240:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697888.874514:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697889.898525:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697890.922522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697891.946504:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697892.970590:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697893.098510:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098519:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697893.098523:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697893.098534:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.098565:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697893.098572:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697893.098624:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145962368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697893.098653:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697893.098657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697893.098661:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.098663:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697893.098667:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.098676:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697893.098690:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145962496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697893.098700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098704:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697893.098708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697893.098709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145962368:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697893.098714:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.098720:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697893.098724:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.098729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697893.098744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145962624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697893.098753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098757:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697893.098759:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697893.098763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.098775:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697893.098782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697893.098782:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.098791:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697893.098805:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145962752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697893.098814:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098817:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697893.098819:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697893.098824:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.098828:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697893.098830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183646d80 x1874531145962368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697893.098835:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.098842:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697893.098862:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145962880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697893.098863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145962368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697893.098887:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098893:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697893.098895:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697893.098896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697893.098903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697893.098906:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00100000:0.0:1787697893.098912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145962368:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697893.098913:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697893.098922:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.098929:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697893.098943:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145963008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697893.098948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962496:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697893.098961:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697893.098964:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697893.098966:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697893.098971:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.098975:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697893.098980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697893.098983:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.098990:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697893.098998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183644380 x1874531145962496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697893.099004:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145963136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697893.099012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145962496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697893.099021:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697893.099023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697893.099030:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697893.099033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962496:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697893.099034:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697893.099041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697893.099044:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.099050:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697893.099054:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697893.099060:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697893.099062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697893.099071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183647b80 x1874531145962624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697893.099077:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145963264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697893.099082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145962624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697893.099102:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697893.099104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697893.099110:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697893.099114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962752:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697893.099114:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697893.099123:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5792 00000100:00080000:1.0:1787697893.099128:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697893.099132:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697893.099136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697893.099138:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697893.099149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183646a00 x1874531145962752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697893.099153:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145963392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697893.099177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145962752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697893.099200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697893.099220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697893.099229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183644e00 x1874531145962880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697893.099236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145962880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145962880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697893.099265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697893.099285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697893.099296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183647100 x1874531145963008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697893.099315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145963008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697893.099336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697893.099395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697893.099423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183645180 x1874531145963136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697893.099431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145963136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697893.099444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697893.099458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697893.099463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183645c00 x1874531145963264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697893.099470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145963264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697893.099485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697893.099494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697893.099501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697893/real 1787697893] req@ffff8f5183647480 x1874531145963392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697893.099511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145963392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697893.099519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697893.099522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697893.099525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145963392:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697893.994564:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697895.018530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697896.042606:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697897.066555:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697898.090543:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697898.218495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697898.218504:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697898.218508:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.218517:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.218528:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218535:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697898.218566:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145963520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697898.218592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697898.218595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697898.218597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.218601:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.218605:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218609:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697898.218618:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697898.218630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145963520:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697898.218631:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145963648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697898.218650:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697898.218653:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697898.218654:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.218658:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.218661:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218665:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697898.218670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697898.218680:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145963776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697898.218687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697898.218694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697898.218697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697898.218699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.218704:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.218708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218711:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697898.218715:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697898.218725:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145963904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697898.218730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704ed80 x1874531145963520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697898.218749:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697898.218752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145963520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697898.218753:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697898.218757:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.218766:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:0.0:1787697898.218767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697898.218773:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218777:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697898.218779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.218784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145963520:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787697898.218785:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697898.218798:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145964032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697898.218803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145963648:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697898.218815:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697898.218819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697898.218821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697898.218823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697898.218829:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.218833:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697898.218836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704ea00 x1874531145963648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697898.218841:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697898.218850:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697898.218857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145963648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.218869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.218875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787697898.218876:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145964160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697898.218880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145963648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697898.218892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145963776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697898.218894:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697898.218901:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697898.218903:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.218908:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.218912:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218918:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697898.218919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787697898.218926:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697898.218934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704e300 x1874531145963776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697898.218940:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145964288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697898.218955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145963776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697898.218968:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697898.218972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697898.218973:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697898.218978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697898.218978:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697898.218984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145963776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697898.218988:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00100000:0.0:1787697898.218992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145963904:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697898.218992:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.218997:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697898.219002:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697898.219012:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145964416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697898.219018:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697898.219022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697898.219022:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697898.219026:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697898.219030:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5797 00000100:00080000:2.0:1787697898.219033:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697898.219035:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697898.219039:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697898.219042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704f480 x1874531145963904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697898.219048:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145964544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697898.219056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145963904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.219062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.219064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.219066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145963904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697898.219079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145964032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697898.219092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697898.219098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704d500 x1874531145964032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697898.219104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145964032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.219109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.219111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.219112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145964032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697898.219115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697898.219123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697898.219128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704d180 x1874531145964160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697898.219134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145964160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.219139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.219141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.219142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697898.219145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697898.219152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697898.219157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704dc00 x1874531145964288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697898.219163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145964288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.219167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.219168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.219170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697898.219173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697898.219180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697898.219185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704ed80 x1874531145964416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697898.219190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145964416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.219195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.219196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.219199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697898.219202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697898.219210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697898.219214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697898/real 1787697898] req@ffff8f518704fb80 x1874531145964544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697898.219220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145964544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697898.219224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697898.219226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697898.219227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145964544:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697899.114581:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697900.138513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697901.162609:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697902.186584:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697903.210556:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697903.338445:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697903.338458:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697903.338467:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338477:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338483:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697903.338522:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145964672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697903.338544:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338548:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697903.338550:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697903.338554:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338562:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338570:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697903.338583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145964800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697903.338590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145964672:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697903.338595:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338600:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697903.338602:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697903.338618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338621:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697903.338642:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145964928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697903.338652:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697903.338657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697903.338658:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697903.338661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697903.338671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338678:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338681:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338686:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697903.338701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f5187389c00 x1874531145964672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697903.338702:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145965056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697903.338723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338727:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697903.338729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697903.338733:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00100000:0.0:1787697903.338737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145964672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697903.338738:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338745:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338752:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697903.338758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787697903.338763:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145965184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697903.338765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.338774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145964672:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697903.338781:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697903.338787:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697903.338792:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338795:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338798:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697903.338802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145964800:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787697903.338804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697903.338817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145965312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697903.338828:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697903.338833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697903.338837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697903.338851:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00100000:0.0:1787697903.338853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f518738bb80 x1874531145964800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697903.338859:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338866:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338872:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697903.338876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145964800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697903.338885:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145965440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697903.338889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.338898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697903.338903:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697903.338904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145964800:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697903.338909:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697903.338913:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697903.338917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145964928:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697903.338919:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338926:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338929:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338933:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697903.338943:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145965568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697903.338954:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697903.338957:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697903.338959:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697903.338962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697903.338962:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5802 00000100:00080000:3.0:1787697903.338967:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697903.338971:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697903.338976:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697903.338985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f518738ad80 x1874531145964928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697903.338987:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145965696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697903.339022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145964928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145964928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697903.339044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145965056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697903.339075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697903.339087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f5187389500 x1874531145965056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697903.339105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145965056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145965056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697903.339124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145965184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697903.339141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697903.339151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f518738a300 x1874531145965184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697903.339162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145965184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145965184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697903.339199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697903.339217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697903.339227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f5187389180 x1874531145965312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697903.339238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145965312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697903.339262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697903.339278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697903.339287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f518738aa00 x1874531145965440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697903.339295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145965440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697903.339311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697903.339324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697903.339331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f518738b100 x1874531145965568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697903.339341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145965568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697903.339403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697903.339422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697903.339432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697903/real 1787697903] req@ffff8f518738b800 x1874531145965696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697903.339444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145965696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697903.339450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697903.339453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697903.339456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145965696:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697904.234500:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697905.258479:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697906.282526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697907.306506:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697908.330542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697908.458457:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458464:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697908.458468:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697908.458475:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458483:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697908.458489:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697908.458527:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145965824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697908.458548:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458550:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697908.458552:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697908.458555:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458557:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697908.458559:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697908.458567:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697908.458579:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145965952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697908.458589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458592:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697908.458595:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697908.458598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145965824:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697908.458603:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458607:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697908.458611:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697908.458617:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697908.458627:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145966080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697908.458634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458637:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697908.458638:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697908.458641:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458645:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697908.458648:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697908.458651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697908.458656:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697908.458671:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145966208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697908.458680:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458683:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697908.458685:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697908.458691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183647800 x1874531145965824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697908.458692:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458699:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697908.458705:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697908.458710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145965824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697908.458711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697908.458728:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145966336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697908.458729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.458735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.458741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145965824:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697908.458746:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697908.458750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145965952:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697908.458751:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697908.458755:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697908.458760:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458762:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697908.458765:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697908.458769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697908.458774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697908.458782:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145966464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697908.458786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183646680 x1874531145965952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697908.458794:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458799:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697908.458800:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697908.458802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145965952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697908.458806:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458812:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697908.458816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697908.458817:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697908.458823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697908.458824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697908.458828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145965952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697908.458834:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145966592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697908.458841:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697908.458843:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697908.458844:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697908.458846:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00080000:1.0:1787697908.458848:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697908.458851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145966080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697908.458854:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697908.458859:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697908.458872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697908.458874:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145966720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697908.458881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183644000 x1874531145966080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697908.458894:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697908.458896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145966080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697908.458901:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697908.458906:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697908.458907:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.458913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697908.458917:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5808 00000100:00100000:0.0:1787697908.458920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145966080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697908.458923:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697908.458931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145966208:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697908.458932:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697908.458940:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697908.458955:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145966848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697908.458961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697908.458979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183644700 x1874531145966208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697908.458991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145966208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697908.458997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.458999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.459001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145966208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697908.459004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145966336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697908.459018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697908.459024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183647480 x1874531145966336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697908.459030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145966336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697908.459035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.459036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.459038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145966336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697908.459041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697908.459049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697908.459054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183645c00 x1874531145966464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697908.459060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145966464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697908.459065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.459066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.459068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697908.459071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697908.459080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697908.459084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183645180 x1874531145966592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697908.459090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145966592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697908.459094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.459096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.459097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697908.459108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697908.459116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697908.459121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183647100 x1874531145966720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697908.459127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145966720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697908.459132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.459134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.459137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697908.459142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697908.459158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697908.459168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697908/real 1787697908] req@ffff8f5183644e00 x1874531145966848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697908.459179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145966848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697908.459185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697908.459187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697908.459189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145966848:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697909.354543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697910.378497:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697911.402609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697912.426521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697913.450554:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697913.578503:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578513:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697913.578517:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578527:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578540:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578548:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697913.578597:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145966976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697913.578640:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578645:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578649:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578662:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697913.578673:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145967104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578681:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578684:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697913.578685:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578691:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00100000:0.0:1787697913.578693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145966976:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697913.578694:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578698:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578704:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697913.578711:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145967232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578716:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578719:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697913.578720:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578722:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578725:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578727:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697913.578736:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145967360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697913.578745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578747:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578750:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578752:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697913.578760:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145967488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578766:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578768:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697913.578769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578772:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578775:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697913.578779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697913.578780:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578785:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697913.578793:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145967616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578800:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578802:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697913.578803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578806:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578808:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578810:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578813:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697913.578819:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145967744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578824:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697913.578825:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697913.578826:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578829:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578831:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697913.578834:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697913.578837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704d880 x1874531145966976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697913.578840:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697913.578855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145967872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697913.578867:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697913.578870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145966976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697913.578871:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697913.578875:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697913.578883:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5813 00000100:00080000:2.0:1787697913.578889:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697913.578891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697913.578892:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697913.578900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697913.578901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.578907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145966976:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697913.578911:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145968000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697913.578940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697913.578962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.578974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704c700 x1874531145967104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.578987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145967104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697913.579014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697913.579035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704c380 x1874531145967232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145967232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697913.579079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697913.579096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704c000 x1874531145967360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145967360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697913.579130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697913.579148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704e680 x1874531145967488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145967488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145967488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697913.579182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145967616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697913.579214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704ce00 x1874531145967616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145967616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145967616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697913.579242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145967744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697913.579254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704df80 x1874531145967744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145967744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145967744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697913.579294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145967872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697913.579313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704f800 x1874531145967872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145967872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145967872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697913.579405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145968000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697913.579462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697913.579470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697913/real 1787697913] req@ffff8f518704fb80 x1874531145968000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697913.579481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145968000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697913.579488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697913.579490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697913.579493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145968000:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697914.474639:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697915.498583:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697916.522513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697917.546519:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697918.570598:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697918.698463:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698470:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697918.698473:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698482:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697918.698500:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697918.698544:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145968128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698570:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698574:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697918.698576:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698589:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698597:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697918.698601:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697918.698609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697918.698621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145968128:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697918.698623:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145968256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698645:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698648:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697918.698649:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698652:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697918.698657:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697918.698660:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697918.698668:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145968384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698673:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787697918.698679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697918.698679:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698684:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698687:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697918.698689:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697918.698692:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697918.698699:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145968512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697918.698708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698713:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697918.698715:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697918.698718:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697918.698724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704ed80 x1874531145968128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697918.698727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145968640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698755:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697918.698759:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698762:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697918.698768:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697918.698769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145968128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697918.698771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697918.698779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145968768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697918.698790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697918.698792:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697918.698796:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697918.698800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697918.698800:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698806:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697918.698808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145968128:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697918.698811:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697918.698816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697918.698823:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145968896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697918.698833:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697918.698836:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697918.698836:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697918.698839:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698841:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697918.698843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968256:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697918.698844:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697918.698854:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697918.698870:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145969024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697918.698872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697918.698884:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697918.698885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704dc00 x1874531145968256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697918.698892:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697918.698896:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697918.698900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145968256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697918.698903:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5818 00000100:00080000:2.0:1787697918.698911:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697918.698912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.698918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697918.698919:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697918.698921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968256:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787697918.698926:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697918.698928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697918.698941:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145969152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697918.698946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.698956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704d180 x1874531145968384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.698974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145968384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.698982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.698985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.698988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697918.698994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697918.699013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.699023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704d500 x1874531145968512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.699034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145968512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.699042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.699045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.699048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697918.699053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697918.699065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.699074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704f480 x1874531145968640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.699084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145968640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.699092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.699095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.699098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145968640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697918.699119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145968768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697918.699138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.699148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704e300 x1874531145968768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.699158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145968768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.699166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.699169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.699172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145968768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697918.699187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145968896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697918.699200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.699206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704ea00 x1874531145968896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.699213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145968896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.699221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.699223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.699225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145968896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697918.699228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145969024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697918.699237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.699242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704ed80 x1874531145969024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.699247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145969024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.699252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.699253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.699255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145969024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697918.699258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145969152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697918.699266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697918.699271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697918/real 1787697918] req@ffff8f518704fb80 x1874531145969152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697918.699277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145969152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697918.699281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697918.699283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697918.699285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145969152:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697919.594534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697920.618598:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697921.642480:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697922.666507:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697923.690500:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697923.818519:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818527:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697923.818529:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697923.818537:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:3.0:1787697923.818547:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818552:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697923.818583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145969280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818602:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818604:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697923.818605:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697923.818608:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:3.0:1787697923.818610:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818619:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697923.818627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145969408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818633:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818636:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697923.818637:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697923.818641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145969280:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697923.818642:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:3.0:1787697923.818647:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818651:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818654:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697923.818661:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145969536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818666:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818669:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697923.818670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697923.818672:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:3.0:1787697923.818674:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818676:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818679:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697923.818689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145969664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697923.818698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697923.818698:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697923.818699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697923.818705:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:3.0:1787697923.818707:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818710:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818713:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697923.818720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145969792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818726:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697923.818730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697923.818733:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00100000:0.0:1787697923.818736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f518738a680 x1874531145969280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697923.818736:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818740:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697923.818756:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145969920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697923.818760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145969280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697923.818768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697923.818777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697923.818780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697923.818781:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:0.0:1787697923.818785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787697923.818785:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818790:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697923.818792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145969280:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787697923.818794:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697923.818803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145970048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818807:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818810:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697923.818812:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697923.818815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969408:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697923.818818:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5823 00000100:00080000:3.0:1787697923.818820:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818822:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697923.818835:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145970176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697923.818843:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697923.818845:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697923.818846:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697923.818850:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5823 00000800:00000100:0.0:1787697923.818852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697923.818853:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697923.818857:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697923.818860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697923.818871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f5187388000 x1874531145969408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697923.818873:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145970304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697923.818907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145969408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.818916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.818920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.818925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697923.818932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697923.818951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.818962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f518738b480 x1874531145969536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.818973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145969536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.818981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.818984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.818987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697923.818992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697923.819006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.819032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f5187388e00 x1874531145969664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.819040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145969664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.819044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.819046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.819048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697923.819052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697923.819064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.819069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f5187389f80 x1874531145969792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.819076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145969792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.819081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.819083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.819084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145969792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697923.819087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145969920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697923.819098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.819106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f518738b800 x1874531145969920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.819116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145969920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.819124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.819127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.819130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145969920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697923.819136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145970048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697923.819152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.819167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f518738b100 x1874531145970048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.819174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145970048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.819180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.819183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.819187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145970048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697923.819205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145970176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697923.819234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.819246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f518738aa00 x1874531145970176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.819255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145970176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.819261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.819264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.819267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145970176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697923.819272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145970304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697923.819287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697923.819295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697923/real 1787697923] req@ffff8f5187389180 x1874531145970304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697923.819304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145970304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697923.819310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697923.819312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697923.819315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145970304:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697924.714567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697925.738541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697926.762542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697927.786587:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697928.810558:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697928.938480:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938490:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697928.938494:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938504:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938516:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938523:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697928.938564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145970432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697928.938596:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938599:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697928.938601:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938606:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938610:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938614:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938624:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697928.938639:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145970560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697928.938649:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697928.938653:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938661:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938665:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697928.938672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145970688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697928.938683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145970432:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697928.938683:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938690:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697928.938692:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938695:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938704:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938708:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697928.938718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145970816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697928.938724:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938727:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697928.938728:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938731:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938733:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938736:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697928.938745:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145970944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697928.938751:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938753:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697928.938755:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938757:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938759:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938761:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938765:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697928.938772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145971072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697928.938780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697928.938790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697928.938794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938800:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938803:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938807:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938812:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697928.938828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145971200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697928.938833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704f800 x1874531145970432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697928.938849:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938861:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697928.938864:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938872:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938876:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697928.938881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145970432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697928.938892:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697928.938894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787697928.938899:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697928.938902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.938908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145970432:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697928.938917:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145971328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697928.938921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970560:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697928.938956:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697928.938959:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697928.938961:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697928.938966:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5828 00000100:00080000:2.0:1787697928.938971:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697928.938974:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697928.938979:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697928.938990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697928.938992:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145971456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697928.939026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704df80 x1874531145970560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145970560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697928.939076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697928.939091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704ce00 x1874531145970688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145970688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697928.939137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697928.939155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704e680 x1874531145970816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145970816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697928.939195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697928.939212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704c000 x1874531145970944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145970944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145970944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697928.939252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697928.939271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704c380 x1874531145971072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145971072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697928.939312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697928.939326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704c700 x1874531145971200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145971200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697928.939351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697928.939385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704d880 x1874531145971328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145971328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697928.939425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697928.939441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697928.939451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff8f518704fb80 x1874531145971456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697928.939462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145971456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697928.939469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697928.939472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697928.939475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145971456:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697929.834585:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697930.858555:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697931.882540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697932.906618:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697933.930541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697934.058505:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058518:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697934.058523:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058536:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058550:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058558:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697934.058605:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145971584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697934.058644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058648:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697934.058650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058664:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697934.058688:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145971712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697934.058698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145971584:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697934.058702:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697934.058709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058713:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058715:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058718:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058720:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697934.058727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145971840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697934.058733:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058735:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697934.058736:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058738:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058740:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058742:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697934.058751:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145971968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697934.058757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697934.058760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058762:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058766:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058769:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058771:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697934.058779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697934.058780:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145972096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697934.058787:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058790:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697934.058791:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058794:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058796:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058799:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697934.058808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145972224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697934.058813:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058815:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697934.058816:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058818:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058821:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058823:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697934.058827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704ed80 x1874531145971584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697934.058828:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697934.058865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145971584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697934.058867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145972352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697934.058886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.058893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.058902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145971584:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697934.058902:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058908:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697934.058912:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058916:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058919:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058922:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058926:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697934.058935:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145972480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697934.058940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145971712:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697934.058943:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697934.058947:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697934.058949:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697934.058951:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5833 00000100:00080000:2.0:1787697934.058954:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697934.058957:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697934.058962:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697934.058967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697934.058976:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145972608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697934.058984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704ea00 x1874531145971712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145971712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145971712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697934.059025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145971840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697934.059041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704e300 x1874531145971840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145971840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145971840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697934.059072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145971968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697934.059082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704f480 x1874531145971968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145971968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145971968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697934.059105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145972096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697934.059115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704d500 x1874531145972096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145972096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145972096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697934.059138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697934.059147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704d180 x1874531145972224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145972224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697934.059170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697934.059179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704dc00 x1874531145972352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145972352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697934.059210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697934.059219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704ed80 x1874531145972480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145972480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697934.059242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697934.059251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697934.059256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697934/real 1787697934] req@ffff8f518704fb80 x1874531145972608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697934.059262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145972608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697934.059267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697934.059269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697934.059270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145972608:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697934.954574:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697935.978611:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697937.002540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697938.026562:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697939.050510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697939.178526:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697939.178538:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697939.178541:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178552:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178563:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178570:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697939.178616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145972736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178649:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697939.178654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697939.178656:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178661:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178665:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178670:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178679:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697939.178690:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145972864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178699:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697939.178700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145972736:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697939.178704:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697939.178707:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178711:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178713:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178715:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178718:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697939.178728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145972992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697939.178737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697939.178738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178742:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178744:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178746:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178771:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697939.178779:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145973120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178786:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697939.178788:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697939.178789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178791:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697939.178799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787697939.178800:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697939.178818:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145973248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178828:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697939.178831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697939.178833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178837:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178842:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178847:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697939.178856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f518738a300 x1874531145972736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697939.178868:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145973376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178878:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697939.178880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145972736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697939.178883:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697939.178886:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697939.178897:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697939.178901:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:0.0:1787697939.178903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.178908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145972736:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787697939.178909:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178915:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178919:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697939.178928:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145973504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697939.178932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145972864:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787697939.178939:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697939.178943:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697939.178945:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178948:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178951:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178953:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.178957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697939.178959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787697939.178967:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145973632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697939.178974:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697939.178977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f5187389500 x1874531145972864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697939.178977:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697939.178980:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697939.178985:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5838 00000100:00080000:3.0:1787697939.178989:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697939.178995:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697939.179000:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697939.179002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145972864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697939.179011:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145973760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697939.179013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145972864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697939.179038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145972992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697939.179060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f518738ad80 x1874531145972992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145972992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145972992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697939.179102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145973120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697939.179121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f518738bb80 x1874531145973120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145973120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145973120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697939.179162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145973248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697939.179180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f5187389c00 x1874531145973248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145973248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145973248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697939.179234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697939.179253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f5187388a80 x1874531145973376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145973376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697939.179308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697939.179327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f5187388380 x1874531145973504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145973504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697939.179423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697939.179442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f518738a300 x1874531145973632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145973632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697939.179489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697939.179507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697939.179518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697939/real 1787697939] req@ffff8f5187389180 x1874531145973760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697939.179528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145973760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697939.179535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697939.179539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697939.179542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145973760:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697940.074609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697941.098589:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697942.122581:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697943.146598:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697944.170591:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697944.298489:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298499:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697944.298503:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298514:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298526:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298532:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697944.298582:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145973888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697944.298618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298621:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298624:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298627:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298642:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697944.298653:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145974016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298661:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298676:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697944.298678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145973888:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697944.298678:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298684:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298689:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298691:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697944.298706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145974144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697944.298717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298719:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298724:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298727:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697944.298734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145974272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298739:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697944.298742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298745:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298748:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298750:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298752:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697944.298760:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145974400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298766:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697944.298767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697944.298771:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697944.298773:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298776:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298778:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298780:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298784:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697944.298792:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145974528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298802:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298810:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697944.298812:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298816:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298821:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697944.298823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704d880 x1874531145973888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697944.298828:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298844:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697944.298845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145973888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.298855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697944.298859:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145974656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697944.298861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.298868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145973888:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697944.298880:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697944.298885:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697944.298887:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298892:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00100000:0.0:1787697944.298897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974016:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697944.298897:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298907:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298914:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697944.298925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697944.298929:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145974784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697944.298940:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697944.298944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704c700 x1874531145974016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697944.298944:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697944.298947:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697944.298952:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5843 00000100:00080000:2.0:1787697944.298955:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697944.298960:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697944.298968:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697944.298979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145974016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697944.298979:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145974912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697944.299005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697944.299022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697944.299052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704c380 x1874531145974144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145974144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697944.299094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697944.299112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704c000 x1874531145974272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145974272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697944.299153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697944.299171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704e680 x1874531145974400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145974400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145974400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697944.299212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697944.299228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704ce00 x1874531145974528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145974528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697944.299269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697944.299297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704df80 x1874531145974656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145974656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697944.299395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697944.299416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704f800 x1874531145974784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145974784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697944.299459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697944.299476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697944.299486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697944/real 1787697944] req@ffff8f518704d880 x1874531145974912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697944.299496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145974912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697944.299505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697944.299508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697944.299511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145974912:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697945.194572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697946.218555:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697947.242676:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697948.266626:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697949.290539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697949.418519:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418533:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697949.418538:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418551:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418564:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418573:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697949.418640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145975040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.418671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697949.418677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418682:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418685:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418689:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.418700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697949.418716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145975168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697949.418732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145975040:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697949.418735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418740:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697949.418742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418747:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418750:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418753:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.418758:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697949.418768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145975296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.418777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418781:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697949.418783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418786:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418789:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418792:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.418797:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697949.418807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145975424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.418815:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697949.418819:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418830:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418835:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418840:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.418845:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697949.418857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697949.418859:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145975552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.418872:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418875:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697949.418876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418880:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418884:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418887:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.418892:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697949.418896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697949.418905:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145975680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.418913:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.418916:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697949.418917:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418920:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418923:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697949.418924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975424:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697949.418928:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.418934:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697949.418942:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145975808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.418953:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697949.418957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975552:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697949.418965:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697949.418967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.418971:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.418974:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697949.418977:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697949.418981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145975680:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787697949.418982:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697949.418995:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145975936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697949.419002:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697949.419004:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697949.419006:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697949.419009:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5849 00000100:00080000:2.0:1787697949.419011:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697949.419015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145975808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697949.419015:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697949.419021:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697949.419030:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145976064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697949.419056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145975936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697949.419081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145976064:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697950.314544:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697951.338502:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697952.362541:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787697952.490563:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787697952.490607:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787697952.490624:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490679:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490691:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490700:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490708:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490714:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490721:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490729:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787697952.490735:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787697952.490810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704d880 x1874531145976064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.490835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145976064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.490847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.490852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.490856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145976064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.490863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704e300 x1874531145975808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.490875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145975808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.490881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.490885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.490888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145975808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697952.490893:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704ea00 x1874531145975936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.490905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145975936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.490913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.490916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.490919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145975936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.490925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704d500 x1874531145975552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.490937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145975552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.490945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.490947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.490949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.490954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704f480 x1874531145975680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.490961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145975680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.490966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.490969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.490971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145975680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697952.490978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704ed80 x1874531145975168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.490989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145975168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.491011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.491015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.491018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697952.491025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704dc00 x1874531145975296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.491036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145975296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.491043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.491046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.491049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697952.491055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704d180 x1874531145975424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.491066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145975424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.491071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.491073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.491075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145975424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697952.491078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697949/real 1787697952] req@ffff8f518704fb80 x1874531145975040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697952.491085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145975040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697952.491090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697952.491091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697952.491093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145975040:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787697953.386534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697954.410525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697954.538453:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538463:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697954.538466:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538475:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538485:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538503:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787697954.538535:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145976192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538565:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697954.538570:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538574:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538577:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538581:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538590:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697954.538603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145976320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538615:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697954.538617:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538620:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697954.538642:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145976448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538650:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697954.538654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538657:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538660:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538663:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697954.538676:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145976576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538686:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697954.538688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538695:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538698:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538702:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697954.538711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145976704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538719:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538722:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697954.538724:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538728:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538730:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538733:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697954.538747:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145976832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538755:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697954.538760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538769:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538772:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538776:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787697954.538790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145976960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697954.538807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538811:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538814:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538816:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538820:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697954.538827:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145977088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787697954.538834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697954.538837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697954.538838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697954.538841:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5854 00000100:00080000:3.0:1787697954.538844:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787697954.538848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787697954.538852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787697954.538862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145977216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697954.538945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145976192:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787697954.539003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f518738aa00 x1874531145976192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145976192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145976192:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787697954.539075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697954.539092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f518738b100 x1874531145976320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145976320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697954.539129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697954.539150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f518738b800 x1874531145976448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145976448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697954.539184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697954.539199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f5187389f80 x1874531145976576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145976576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697954.539241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697954.539257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f5187388e00 x1874531145976704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145976704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145976704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697954.539298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145976832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697954.539314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f518738b480 x1874531145976832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145976832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145976832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697954.539352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145976960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697954.539395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f5187388000 x1874531145976960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145976960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145976960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697954.539438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145977088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697954.539454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f518738a680 x1874531145977088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145977088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145977088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697954.539506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145977216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697954.539524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697954.539533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697954/real 1787697954] req@ffff8f5187389180 x1874531145977216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697954.539543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145977216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697954.539556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697954.539558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697954.539561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145977216:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697955.434551:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697956.458548:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697957.482541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697958.506514:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697959.530496:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697959.658516:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658524:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697959.658527:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658535:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658550:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658556:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697959.658592:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145977344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697959.658620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658623:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658626:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658629:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697959.658646:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145977472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658652:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658654:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697959.658656:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658658:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658660:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658662:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697959.658669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145977344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697959.658673:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145977600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658680:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658683:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697959.658684:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658686:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658688:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658690:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697959.658700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145977728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697959.658708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658712:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658714:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697959.658722:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145977856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658726:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697959.658730:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697959.658734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697959.658734:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658739:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658742:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658746:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697959.658751:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145977984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697959.658760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658762:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658766:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697959.658775:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145978112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658781:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697959.658785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704f800 x1874531145977344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697959.658786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697959.658789:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658797:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658803:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658809:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697959.658810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145977344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697959.658816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697959.658824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.658829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.658834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145977344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787697959.658840:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145978240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697959.658855:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697959.658857:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697959.658860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977472:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697959.658861:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697959.658869:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5859 00000100:00080000:2.0:1787697959.658874:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697959.658877:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697959.658882:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697959.658888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697959.658896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145978368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697959.658912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704df80 x1874531145977472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.658926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145977472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.658934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.658938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.658941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697959.658961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697959.658977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.658986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704ce00 x1874531145977600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.658997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145977600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697959.659014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697959.659028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.659037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704e680 x1874531145977728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.659046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145977728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697959.659064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697959.659078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.659086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704c000 x1874531145977856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.659096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145977856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145977856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697959.659113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145977984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697959.659127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.659144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704c380 x1874531145977984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.659154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145977984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145977984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697959.659173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145978112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697959.659187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.659207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704c700 x1874531145978112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.659223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145978112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145978112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697959.659242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145978240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697959.659257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.659265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704fb80 x1874531145978240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.659275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145978240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145978240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697959.659299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145978368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697959.659312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697959.659320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697959/real 1787697959] req@ffff8f518704d180 x1874531145978368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697959.659329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145978368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697959.659341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697959.659343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697959.659346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145978368:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697960.554528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697961.578553:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697962.602517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697963.626556:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697964.650534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697964.778484:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778492:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697964.778496:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697964.778505:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778515:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778521:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697964.778554:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145978496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697964.778579:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697964.778589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697964.778593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778600:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697964.778609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697964.778619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145978496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697964.778622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145978624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697964.778634:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778637:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697964.778638:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697964.778642:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778645:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778649:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697964.778654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697964.778664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145978752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697964.778672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778675:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697964.778677:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697964.778681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697964.778685:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778690:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778694:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697964.778699:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697964.778708:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145978880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697964.778716:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778719:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697964.778721:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697964.778726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183646a00 x1874531145978496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697964.778728:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778733:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778740:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697964.778746:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697964.778751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145978496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697964.778758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145979008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697964.778760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697964.778771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697964.778774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697964.778778:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787697964.778782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145978496:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697964.778783:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697964.778791:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778795:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778799:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697964.778804:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697964.778811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145978624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697964.778816:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145979136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697964.778826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778830:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697964.778832:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697964.778836:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5864 00000800:00000100:0.0:1787697964.778839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697964.778842:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778848:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697964.778852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183647b80 x1874531145978624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697964.778853:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697964.778869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145979264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697964.778877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145978624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697964.778881:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697964.778885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697964.778886:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697964.778890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697964.778891:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697964.778894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145978624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697964.778899:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00100000:0.0:1787697964.778903:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145978752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697964.778904:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697964.778911:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697964.778916:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697964.778921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.778933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183644380 x1874531145978752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697964.778934:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145979392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697964.778953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145978752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697964.778956:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697964.778966:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697964.778967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697964.778969:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697964.778972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697964.778977:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5864 00000100:00080000:1.0:1787697964.778981:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697964.778983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145978752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697964.778987:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697964.778991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145978880:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697964.778995:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697964.779009:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145979520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697964.779012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.779022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183646d80 x1874531145978880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697964.779031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145978880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697964.779036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697964.779038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697964.779040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145978880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697964.779045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145979008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697964.779055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.779061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183644e00 x1874531145979008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697964.779067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145979008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697964.779071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697964.779073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697964.779075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145979008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697964.779085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697964.779094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.779099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183647100 x1874531145979136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697964.779105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145979136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697964.779110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697964.779111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697964.779113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697964.779116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697964.779127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.779136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183645180 x1874531145979264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697964.779152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145979264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697964.779168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697964.779171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697964.779174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697964.779180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697964.779199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.779209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183645c00 x1874531145979392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697964.779220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145979392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697964.779225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697964.779226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697964.779228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697964.779232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697964.779243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697964.779249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697964/real 1787697964] req@ffff8f5183647480 x1874531145979520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697964.779260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145979520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697964.779273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697964.779276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697964.779278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145979520:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697965.674524:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697966.698490:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697967.722529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697968.746540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697969.770541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697969.898454:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898464:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697969.898468:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898477:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898487:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898494:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697969.898530:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145979648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697969.898559:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898562:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697969.898571:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898578:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898582:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898585:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898594:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697969.898604:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145979776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697969.898613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898616:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697969.898618:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898621:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898625:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898628:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898632:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697969.898641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145979904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697969.898650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145979648:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697969.898651:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898657:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697969.898660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898664:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898667:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898671:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898675:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697969.898685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145980032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697969.898693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898696:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697969.898698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898701:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898715:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898719:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697969.898728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697969.898738:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145980160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697969.898747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697969.898752:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898755:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898759:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898763:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898767:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697969.898772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183644700 x1874531145979648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.898780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145980288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697969.898791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697969.898795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145979648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697969.898796:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697969.898799:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898806:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898812:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697969.898814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697969.898817:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697969.898821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697969.898823:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697969.898829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145979648:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697969.898835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145980416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697969.898840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145979776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697969.898851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697969.898856:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697969.898861:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:1.0:1787697969.898867:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697969.898873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697969.898874:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898881:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697969.898891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183644000 x1874531145979776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.898910:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145980544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697969.898911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145979776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697969.898936:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697969.898944:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697969.898948:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697969.898949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697969.898953:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5869 00000100:00080000:0.0:1787697969.898954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.898961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145979776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697969.898961:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.898969:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697969.898974:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697969.898984:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145980672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697969.898991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145979904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697969.899022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183646680 x1874531145979904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145979904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145979904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697969.899064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145980032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697969.899080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183647800 x1874531145980032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145980032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145980032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697969.899139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145980160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697969.899155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183647480 x1874531145980160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145980160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145980160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697969.899190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697969.899206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183645c00 x1874531145980288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145980288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697969.899261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697969.899279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183645180 x1874531145980416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145980416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697969.899317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697969.899334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183647100 x1874531145980544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145980544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697969.899449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697969.899470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697969.899486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff8f5183644e00 x1874531145980672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697969.899497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145980672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.899504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.899508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697969.899511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145980672:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697970.794533:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697971.818552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697972.842502:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697973.866522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697974.890532:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697975.018462:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.018473:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697975.018477:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697975.018485:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.018494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697975.018500:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697975.018579:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145980800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697975.018600:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.018603:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697975.018604:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697975.018607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.018611:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697975.018615:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697975.018623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697975.018636:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145980928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697975.018642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145980800:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697975.018660:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.018677:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697975.018678:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697975.018682:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.018685:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697975.018689:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697975.018693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697975.018697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787697975.018704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145981056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697975.018711:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.018713:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697975.018715:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697975.018717:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.018720:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697975.018722:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697975.018726:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697975.018732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704dc00 x1874531145980800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697975.018738:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145981184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697975.018749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145980800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697975.018751:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.018760:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697975.018764:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697975.018766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697975.018771:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:0.0:1787697975.018772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697975.018778:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697975.018779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145980800:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697975.018786:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697975.018798:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697975.018802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145980928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697975.018815:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145981312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697975.018820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697975.018831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704ed80 x1874531145980928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697975.018832:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697975.018841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145980928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697975.018848:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697975.018855:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697975.018860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.018864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.018866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145980928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697975.018869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145981056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697975.018873:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.018878:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697975.018885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697975.018892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704f480 x1874531145981056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697975.018898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145981056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697975.018899:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697975.018907:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697975.018910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.018915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.018920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145981056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697975.018924:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145981440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697975.018930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145981184:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697975.018939:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.018944:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697975.018946:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697975.018951:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.018955:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787697975.018958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697975.018963:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697975.018969:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697975.018975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704d500 x1874531145981184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697975.018983:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145981568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697975.018993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145981184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697975.019004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.019011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.019017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145981184:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697975.019020:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697975.019023:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697975.019024:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697975.019032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145981312:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697975.019038:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00080000:2.0:1787697975.019043:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697975.019046:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697975.019050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787697975.019057:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697975.019059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704ea00 x1874531145981312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697975.019069:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145981696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697975.019071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145981312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697975.019094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697975.019094:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697975.019101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697975.019101:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697975.019105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145981312:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787697975.019105:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697975.019110:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5874 00000100:00100000:0.0:1787697975.019113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981440:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697975.019119:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697975.019123:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697975.019128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787697975.019129:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697975.019151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704e300 x1874531145981440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697975.019155:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145981824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697975.019169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145981440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697975.019178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.019180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.019182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697975.019186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697975.019200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697975.019206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704d880 x1874531145981568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697975.019213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145981568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697975.019217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.019218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.019220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697975.019228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697975.019237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697975.019242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704d500 x1874531145981696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697975.019248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145981696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697975.019252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.019253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.019255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697975.019258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697975.019267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697975.019271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697975/real 1787697975] req@ffff8f518704f480 x1874531145981824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697975.019279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145981824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697975.019287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697975.019290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697975.019293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145981824:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697975.914527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697976.938548:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697977.962594:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697978.986550:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697980.010473:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697980.138479:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697980.138486:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697980.138489:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697980.138502:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:1.0:1787697980.138511:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697980.138519:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697980.138557:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145981952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697980.138579:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697980.138584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697980.138589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697980.138617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145981952:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697980.138625:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:1.0:1787697980.138629:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697980.138632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697980.138646:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697980.138656:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145982080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697980.138664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697980.138665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697980.138669:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697980.138671:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697980.138676:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:1.0:1787697980.138682:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697980.138685:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697980.138690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697980.138703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183646d80 x1874531145981952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697980.138705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145982208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697980.138733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697980.138736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697980.138738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145981952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697980.138739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697980.138748:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:0.0:1787697980.138750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697980.138753:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697980.138755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.138760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145981952:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697980.138762:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697980.138769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697980.138780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697980.138782:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145982336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697980.138795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697980.138800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697980.138800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697980.138805:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697980.138811:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00100000:0.0:1787697980.138814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183644380 x1874531145982080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697980.138816:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697980.138824:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697980.138827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145982080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697980.138836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697980.138838:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697980.138841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.138846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697980.138856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697980.138857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145982464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697980.138871:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697980.138875:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697980.138878:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697980.138882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697980.138886:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:1.0:1787697980.138891:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697980.138896:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697980.138899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183647b80 x1874531145982208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697980.138903:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697980.138918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145982208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697980.138922:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145982592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697980.138934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697980.138940:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697980.138941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697980.138947:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697980.138948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982208:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697980.138951:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697980.138958:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:1.0:1787697980.138962:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697980.138965:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697980.138969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982336:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697980.138971:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697980.138987:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145982720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697980.138996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697980.138998:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697980.139002:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697980.139005:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697980.139009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183646a00 x1874531145982336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697980.139011:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00080000:1.0:1787697980.139020:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697980.139024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145982336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697980.139029:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787697980.139033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697980.139037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787697980.139037:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697980.139041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697980.139047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787697980.139067:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145982848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697980.139077:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787697980.139082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697980.139082:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697980.139086:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697980.139092:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5879 00000100:00100000:0.0:1787697980.139093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183646d80 x1874531145982464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697980.139100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145982464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697980.139105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697980.139109:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697980.139110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.139115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145982464:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697980.139119:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697980.139125:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697980.139131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787697980.139141:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145982976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697980.139154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697980.139166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183644e00 x1874531145982592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697980.139176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145982592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697980.139182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697980.139184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.139186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697980.139196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697980.139208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697980.139214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183647100 x1874531145982720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697980.139224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145982720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697980.139228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697980.139230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.139231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697980.139234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697980.139243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697980.139248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183645180 x1874531145982848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697980.139254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145982848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697980.139258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697980.139259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.139261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697980.139263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697980.139271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697980.139283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697980/real 1787697980] req@ffff8f5183645c00 x1874531145982976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697980.139292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145982976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697980.139300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697980.139302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697980.139304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145982976:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787697981.034539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697982.058523:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697983.082514:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697984.106536:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697985.130538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697985.258409:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697985.258419:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697985.258423:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697985.258432:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:1.0:1787697985.258442:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258449:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787697985.258481:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145983104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697985.258504:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697985.258507:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697985.258509:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697985.258513:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:1.0:1787697985.258516:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258520:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697985.258529:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697985.258534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145983104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787697985.258543:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531145983232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697985.258553:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697985.258556:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697985.258558:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697985.258561:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:1.0:1787697985.258574:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258578:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697985.258582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787697985.258584:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697985.258597:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531145983360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697985.258606:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697985.258609:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787697985.258612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183647480 x1874531145983104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258613:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697985.258621:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:1.0:1787697985.258627:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258631:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697985.258634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145983104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697985.258637:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697985.258646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697985.258650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787697985.258653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531145983488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697985.258655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145983104:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787697985.258668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697985.258671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697985.258673:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787697985.258677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983232:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697985.258678:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:1.0:1787697985.258684:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258688:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697985.258692:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787697985.258696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697985.258705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531145983616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697985.258708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183647800 x1874531145983232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258730:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697985.258732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531145983232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697985.258739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697985.258745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697985.258748:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:0.0:1787697985.258751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697985.258754:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697985.258755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983232:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697985.258762:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697985.258764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983360:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787697985.258770:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697985.258779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697985.258784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531145983744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697985.258789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183646680 x1874531145983360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258805:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697985.258809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531145983360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258810:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697985.258813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697985.258821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697985.258821:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:0.0:1787697985.258827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787697985.258827:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697985.258833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983360:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787697985.258835:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697985.258840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983488:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697985.258843:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787697985.258858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787697985.258858:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531145983872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697985.258866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183644000 x1874531145983488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258873:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787697985.258877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531145983488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697985.258882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697985.258888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697985.258891:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00080000:0.0:1787697985.258893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697985.258898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983488:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787697985.258898:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258906:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697985.258911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983616:192.168.201.125@tcp:8:kworker.0 00000008:00100000:1.0:1787697985.258911:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697985.258928:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531145984000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787697985.258938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787697985.258940:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697985.258945:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697985.258947:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697985.258952:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5884 00000100:00100000:0.0:1787697985.258954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183644700 x1874531145983616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697985.258956:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697985.258976:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787697985.258984:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697985.258989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531145983616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697985.258997:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145984128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697985.258999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697985.259005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697985.259011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145983616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697985.259018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145983744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697985.259034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697985.259041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183647480 x1874531145983744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697985.259047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531145983744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697985.259053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697985.259055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697985.259057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145983744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697985.259059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145983872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697985.259073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697985.259078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183645c00 x1874531145983872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697985.259091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531145983872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697985.259098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697985.259100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697985.259103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145983872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697985.259116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145984000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697985.259134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697985.259141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183645180 x1874531145984000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697985.259148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531145984000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697985.259153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697985.259154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697985.259156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145984000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697985.259160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145984128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697985.259170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697985.259175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697985/real 1787697985] req@ffff8f5183647100 x1874531145984128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697985.259181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145984128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697985.259189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697985.259190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697985.259192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145984128:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697986.154532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697987.178544:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697988.202465:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697989.226505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697990.250490:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697990.378452:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697990.378462:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378469:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378478:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697990.378483:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697990.378508:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145984256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697990.378527:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378529:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697990.378530:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378533:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378535:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697990.378537:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378545:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697990.378552:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145984384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697990.378559:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378562:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697990.378563:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378565:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378567:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697990.378571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145984256:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697990.378578:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378582:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697990.378595:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145984512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697990.378606:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378610:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697990.378612:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378615:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378619:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697990.378624:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787697990.378625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697990.378641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145984640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697990.378647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378650:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697990.378651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378654:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697990.378660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704ed80 x1874531145984256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697990.378661:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697990.378676:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145984768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697990.378680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145984256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697990.378688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378695:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697990.378697:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697990.378699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697990.378701:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378704:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787697990.378705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697990.378709:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697990.378711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145984256:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787697990.378714:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697990.378718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697990.378723:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145984896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697990.378729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697990.378732:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378734:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5890 00000800:00000100:0.0:1787697990.378738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697990.378740:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697990.378743:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378745:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697990.378754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145985024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697990.378759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704dc00 x1874531145984384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697990.378762:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378770:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697990.378772:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378777:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00100000:0.0:1787697990.378779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145984384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697990.378780:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697990.378787:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787697990.378802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697990.378803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145985152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697990.378808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787697990.378816:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697990.378818:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787697990.378819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984384:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787697990.378820:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697990.378827:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5890 00000100:00080000:2.0:1787697990.378830:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697990.378832:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697990.378835:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697990.378840:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145985280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697990.378845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697990.378860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.378867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704d180 x1874531145984512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.378893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145984512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.378899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.378901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.378903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697990.378907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697990.378920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.378927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704fb80 x1874531145984640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.378934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145984640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.378938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.378940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.378942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697990.378946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697990.378954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.378959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704c700 x1874531145984768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.378965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145984768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.378969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.378971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.378972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145984768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697990.378975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145984896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697990.378983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.378988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704c380 x1874531145984896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.378995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145984896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.378999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.379001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.379003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145984896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697990.379005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145985024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697990.379013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.379018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704c000 x1874531145985024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.379024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145985024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.379029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.379030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.379032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145985024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697990.379035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145985152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697990.379042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.379047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704e680 x1874531145985152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.379052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145985152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.379056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.379058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.379060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145985152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697990.379070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145985280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697990.379078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697990.379084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697990/real 1787697990] req@ffff8f518704ce00 x1874531145985280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697990.379090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145985280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697990.379095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697990.379096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697990.379098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145985280:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787697991.274547:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787697992.298479:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697993.322534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697994.346537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697995.370525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697995.498436:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498446:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697995.498450:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498461:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498472:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498479:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787697995.498514:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145985408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498539:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498542:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697995.498544:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498549:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498553:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498557:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498565:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697995.498578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145985536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498594:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498598:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787697995.498600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145985408:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697995.498600:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498605:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498610:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498612:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498616:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697995.498626:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145985664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498632:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498634:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697995.498635:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498637:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498640:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498642:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498644:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697995.498650:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145985792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498655:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498657:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697995.498658:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498663:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5895 00000800:00000100:0.0:1787697995.498664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697995.498666:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498669:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498671:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697995.498679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145985920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498684:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498686:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697995.498687:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498689:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498691:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498693:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498696:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787697995.498707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704df80 x1874531145985408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697995.498707:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145986048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697995.498729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498732:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498734:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787697995.498737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145985408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697995.498739:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498744:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787697995.498752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787697995.498754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145986176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787697995.498758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.498764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145985408:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787697995.498776:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498779:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697995.498781:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697995.498786:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498804:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498809:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787697995.498814:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787697995.498817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787697995.498826:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145986304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697995.498836:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697995.498840:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697995.498841:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787697995.498846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787697995.498849:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5895 00000100:00080000:2.0:1787697995.498854:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697995.498858:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787697995.498863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704f800 x1874531145985536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697995.498865:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697995.498881:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145986432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787697995.498883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145985536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.498911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.498916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.498920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697995.498927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697995.498947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.498958:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704ce00 x1874531145985664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.498967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145985664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.498972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.498974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.498976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697995.498980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697995.499009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.499019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704e680 x1874531145985792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.499038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145985792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.499046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.499048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.499051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697995.499056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697995.499074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.499084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704c000 x1874531145985920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.499095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145985920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.499101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.499103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.499107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145985920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697995.499112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697995.499129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.499140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704c380 x1874531145986048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.499148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145986048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.499154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.499157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.499159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697995.499164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787697995.499180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.499190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704c700 x1874531145986176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.499201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145986176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.499209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.499213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.499215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787697995.499238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697995.499256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.499267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704fb80 x1874531145986304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.499278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145986304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.499285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.499288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.499290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787697995.499294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787697995.499307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787697995.499313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697995/real 1787697995] req@ffff8f518704d180 x1874531145986432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697995.499319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145986432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697995.499324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697995.499326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787697995.499327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145986432:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787697996.394553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787697997.418522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697998.442535:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787697999.466561:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698000.490571:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698000.618507:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618521:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698000.618528:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618541:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00080000:3.0:1787698000.618555:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618563:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698000.618613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145986560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698000.618637:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698000.618642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618645:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00080000:3.0:1787698000.618648:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618654:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698000.618662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698000.618672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145986688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698000.618680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698000.618685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618689:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00080000:3.0:1787698000.618693:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618697:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698000.618714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145986560:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698000.618715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698000.618734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145986816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698000.618747:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618750:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698000.618751:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618754:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00080000:3.0:1787698000.618757:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618760:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698000.618763:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698000.618771:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145986944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698000.618777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618779:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698000.618783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698000.618783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618787:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00080000:3.0:1787698000.618790:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618793:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698000.618796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698000.618807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145987072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698000.618816:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618819:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698000.618821:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618826:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00100000:0.0:1787698000.618829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f518738a300 x1874531145986560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698000.618833:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618842:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698000.618848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145986560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698000.618848:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698000.618860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.618864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698000.618868:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145987200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698000.618870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145986560:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698000.618880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145986688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698000.618887:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618891:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698000.618892:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.618897:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5900 00000800:00000100:0.0:1787698000.618901:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698000.618901:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618907:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698000.618913:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698000.618914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f5187388380 x1874531145986688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698000.618927:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145987328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698000.618928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145986688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.618956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698000.618956:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.618962:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698000.618963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698000.618964:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698000.618969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145986688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698000.618970:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00080000:3.0:1787698000.618975:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698000.618977:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698000.618980:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698000.618987:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145987456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698000.618994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145986816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698000.619009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698000.619015:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698000.619019:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698000.619021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f5187388a80 x1874531145986816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698000.619023:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698000.619033:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5900 00000100:00100000:0.0:1787698000.619036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145986816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698000.619040:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698000.619042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698000.619048:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145986816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698000.619059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145986944:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787698000.619059:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698000.619078:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145987584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698000.619081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f5187389c00 x1874531145986944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698000.619099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145986944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.619104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698000.619109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145986944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698000.619114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145987072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698000.619124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f518738bb80 x1874531145987072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698000.619136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145987072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.619141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698000.619144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145987072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698000.619147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698000.619159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f518738ad80 x1874531145987200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698000.619178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145987200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.619187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698000.619193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698000.619199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698000.619218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f5187389500 x1874531145987328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698000.619239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145987328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.619244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698000.619248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698000.619260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698000.619272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f5187389180 x1874531145987456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698000.619284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145987456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.619288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698000.619292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698000.619295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698000.619303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698000.619308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698000/real 1787698000] req@ffff8f518738a680 x1874531145987584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698000.619314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145987584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698000.619323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698000.619325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698000.619326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145987584:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698001.514529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698002.538513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698003.562568:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698004.586595:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698005.610593:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698005.738491:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738501:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698005.738505:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738516:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738527:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738533:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698005.738571:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145987712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698005.738599:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738602:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698005.738604:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738609:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738636:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698005.738646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698005.738656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145987840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698005.738665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145987712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698005.738665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738672:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698005.738674:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738677:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738680:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738682:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698005.738685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698005.738695:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145987968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698005.738701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738703:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698005.738704:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738706:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738709:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738711:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698005.738714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698005.738720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145988096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698005.738725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738728:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698005.738729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738732:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738734:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738736:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698005.738738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698005.738742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698005.738748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145988224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698005.738754:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698005.738758:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738761:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738764:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738766:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698005.738768:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698005.738775:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145988352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698005.738780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738782:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698005.738783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738785:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738788:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738791:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698005.738796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f5187388000 x1874531145987712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698005.738796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698005.738805:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145988480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698005.738811:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738814:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698005.738815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145987712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698005.738816:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738823:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738828:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738830:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698005.738831:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698005.738833:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698005.738839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698005.738843:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145988608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698005.738849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145987712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698005.738851:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698005.738858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698005.738859:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698005.738862:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5905 00000100:00080000:3.0:1787698005.738864:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698005.738866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698005.738869:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698005.738874:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145988736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698005.738890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145987840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698005.738911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.738920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f518738b480 x1874531145987840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.738928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145987840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.738933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.738936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.738938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145987840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698005.738942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145987968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698005.738952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.738957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f5187388e00 x1874531145987968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.738964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145987968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.738968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.738970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.738971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145987968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698005.738975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145988096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698005.738985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.738990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f5187389f80 x1874531145988096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.738996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145988096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.739000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.739002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.739003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145988096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698005.739006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145988224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698005.739015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.739020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f518738b800 x1874531145988224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.739026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145988224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.739037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.739039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.739041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145988224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698005.739044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698005.739055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.739061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f518738b100 x1874531145988352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.739067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145988352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.739072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.739074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.739075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698005.739080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698005.739096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.739106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f518738aa00 x1874531145988480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.739133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145988480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.739140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.739143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.739147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698005.739152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698005.739171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.739179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f518738a680 x1874531145988608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.739186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145988608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.739190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.739192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.739194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698005.739197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698005.739207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698005.739212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698005/real 1787698005] req@ffff8f5187389180 x1874531145988736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698005.739218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145988736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698005.739227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698005.739229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698005.739230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145988736:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698006.634608:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698007.658513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698008.682544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698009.706555:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698010.730560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698010.858431:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858441:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698010.858444:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858454:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858463:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858469:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698010.858496:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145988864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698010.858519:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858522:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698010.858523:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858537:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858540:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858542:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698010.858549:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698010.858559:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145988992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698010.858565:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698010.858569:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858571:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858574:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858578:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698010.858579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145988864:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698010.858581:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698010.858594:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145989120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698010.858605:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858608:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698010.858610:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858614:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858618:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858622:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698010.858627:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698010.858638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145989248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698010.858641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698010.858656:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698010.858662:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858666:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858671:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858675:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698010.858680:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698010.858682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f5187389500 x1874531145988864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698010.858692:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145989376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698010.858700:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858703:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787698010.858705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145988864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698010.858706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858715:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698010.858724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698010.858729:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698010.858732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698010.858736:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698010.858738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145988864:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698010.858748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145988992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698010.858750:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145989504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698010.858759:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858761:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698010.858762:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858765:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858769:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858771:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698010.858775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698010.858776:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698010.858786:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145989632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698010.858792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698010.858793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f518738ad80 x1874531145988992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698010.858796:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698010.858798:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858803:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00080000:3.0:1787698010.858808:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858813:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698010.858816:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698010.858819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145988992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698010.858825:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145989760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698010.858852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698010.858854:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698010.858857:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698010.858859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698010.858860:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698010.858866:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5910 00000100:00100000:0.0:1787698010.858869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145988992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698010.858870:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698010.858876:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698010.858879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698010.858886:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145989888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698010.858898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145989120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698010.858918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.858929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f518738bb80 x1874531145989120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.858940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145989120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.858947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.858955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.858958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145989120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698010.858964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145989248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698010.858980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.858989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f5187389c00 x1874531145989248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.859000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145989248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.859008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.859011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.859014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145989248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698010.859020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145989376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698010.859035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.859044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f5187388a80 x1874531145989376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.859054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145989376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.859061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.859064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.859067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145989376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698010.859072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698010.859089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.859098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f5187388380 x1874531145989504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.859108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145989504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.859116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.859118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.859127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698010.859132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698010.859150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.859160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f518738a300 x1874531145989632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.859171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145989632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.859178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.859180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.859183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698010.859188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698010.859203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.859212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f5187389180 x1874531145989760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.859223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145989760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.859231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.859233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.859236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698010.859241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698010.859256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698010.859265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff8f518738a680 x1874531145989888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.859275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145989888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.859283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.859286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698010.859289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145989888:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698011.754641:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698012.778635:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698013.802572:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698014.826525:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698015.850617:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698015.978461:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978468:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698015.978472:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978479:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978487:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978492:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698015.978527:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145990016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698015.978553:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978557:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698015.978559:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978563:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978567:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978571:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978580:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698015.978593:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145990144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698015.978599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145990016:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698015.978602:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978606:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698015.978607:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978610:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978613:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978617:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698015.978636:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145990272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698015.978646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698015.978651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978655:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978659:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978663:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698015.978690:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145990400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698015.978696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990144:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698015.978705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978711:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698015.978713:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978717:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978727:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698015.978736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698015.978744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145990528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698015.978751:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698015.978755:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978758:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978762:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978764:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698015.978772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698015.978779:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145990656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698015.978789:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698015.978794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978798:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978803:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698015.978806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990528:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698015.978810:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698015.978833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145990656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698015.978840:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145990784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698015.978855:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978866:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698015.978869:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978872:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00080000:3.0:1787698015.978875:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698015.978877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145990784:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698015.978880:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978888:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698015.978900:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145990912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698015.978907:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698015.978910:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698015.978911:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698015.978916:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5915 00000100:00100000:0.0:1787698015.978918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145990912:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698015.978919:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698015.978924:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698015.978929:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698015.978936:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145991040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698015.978961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145991040:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698016.874552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698017.898538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698018.922521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698019.050635:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787698019.050685:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698019.050699:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698019.050748:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787698019.050764:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698019.050773:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698019.050781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f5187389180 x1874531145991040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698019.050783:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698019.050795:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698019.050811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145991040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698019.050816:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698019.050829:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698019.050838:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698019.050843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.050850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.050855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145991040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698019.050864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f518738a680 x1874531145990912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.050877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145990912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.050884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.050888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.050892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145990912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698019.050898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f5187388000 x1874531145990784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.050909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145990784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.050916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.050919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.050922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145990784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698019.050927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f518738b480 x1874531145990656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.050937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145990656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.050945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.050948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.050951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145990656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698019.050957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f5187389f80 x1874531145990400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.050968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145990400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.050975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.050978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.050981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698019.050987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f5187388e00 x1874531145990528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.051012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145990528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.051019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.051022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.051026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698019.051032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f518738b100 x1874531145990144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.051043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145990144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.051060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.051063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.051066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698019.051072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f518738b800 x1874531145990272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.051083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145990272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.051090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.051093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.051096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145990272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698019.051102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698019] req@ffff8f518738aa00 x1874531145990016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698019.051113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145990016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698019.051120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698019.051123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698019.051126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145990016:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787698019.946533:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698020.970551:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698021.098473:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098481:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698021.098484:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098491:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098499:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098505:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698021.098531:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145991168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698021.098551:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098553:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698021.098554:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098557:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098559:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098561:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098569:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698021.098579:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145991296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698021.098588:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698021.098593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698021.098597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145991168:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698021.098599:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098605:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098611:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098615:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698021.098628:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145991424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698021.098638:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098641:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698021.098643:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098646:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098650:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098653:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698021.098662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698021.098673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145991552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698021.098682:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098685:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698021.098686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098690:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098694:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098698:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698021.098704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f518738a300 x1874531145991168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698021.098704:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698021.098717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145991680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698021.098724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145991168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698021.098732:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698021.098735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698021.098737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698021.098740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698021.098741:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698021.098744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145991168:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698021.098750:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098754:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098758:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098762:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698021.098770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698021.098774:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145991808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698021.098784:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098787:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698021.098789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098792:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098796:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698021.098799:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098803:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698021.098810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698021.098815:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145991936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698021.098824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698021.098827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f5187388380 x1874531145991296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698021.098828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698021.098832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098840:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098845:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698021.098847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145991296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698021.098852:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698021.098862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.098868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.098875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698021.098875:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145992064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698021.098887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991424:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698021.098891:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698021.098896:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698021.098898:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698021.098903:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5920 00000100:00080000:3.0:1787698021.098907:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698021.098911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698021.098913:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698021.098920:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698021.098927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f5187388a80 x1874531145991424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698021.098933:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145992192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698021.098948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145991424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.098963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.098967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.098971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698021.098977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698021.098997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698021.099007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f5187389c00 x1874531145991552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698021.099018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145991552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.099026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.099029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.099032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698021.099038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698021.099055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698021.099065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f518738bb80 x1874531145991680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698021.099076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145991680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.099083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.099086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.099090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145991680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698021.099106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145991808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698021.099125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698021.099150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f518738ad80 x1874531145991808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698021.099162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145991808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.099171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.099174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.099176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145991808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698021.099182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145991936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698021.099199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698021.099210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f5187389500 x1874531145991936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698021.099220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145991936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.099227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.099230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.099233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145991936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698021.099238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145992064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698021.099255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698021.099264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f518738a300 x1874531145992064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698021.099274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145992064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.099280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.099282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.099284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145992064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698021.099289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145992192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698021.099305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698021.099314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698021/real 1787698021] req@ffff8f518738aa00 x1874531145992192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698021.099322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145992192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698021.099329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698021.099332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698021.099335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145992192:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698021.994533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698023.018513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698024.042513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698025.066499:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698026.090507:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698026.218443:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698026.218454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698026.218457:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.218467:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218478:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.218485:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698026.218528:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145992320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698026.218556:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698026.218560:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698026.218562:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.218566:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218570:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.218574:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698026.218583:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698026.218598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531145992448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698026.218609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698026.218610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145992320:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698026.218617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698026.218621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.218625:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218629:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.218633:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698026.218637:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698026.218652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531145992576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698026.218662:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698026.218665:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698026.218666:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698026.218672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698026.218676:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218681:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.218686:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698026.218692:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698026.218704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531145992704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698026.218714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698026.218715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f518738b800 x1874531145992320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698026.218724:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698026.218729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.218738:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218745:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698026.218751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145992320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698026.218752:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698026.218761:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698026.218774:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698026.218776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531145992832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698026.218781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.218788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145992320:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698026.218794:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698026.218799:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698026.218801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.218807:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.218817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698026.218818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992448:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698026.218826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698026.218842:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531145992960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698026.218851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698026.218854:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698026.218858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698026.218861:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.218865:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218869:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.218872:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698026.218876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f518738b100 x1874531145992448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698026.218879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698026.218894:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531145993088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698026.218897:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531145992448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698026.218920:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698026.218921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698026.218928:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698026.218931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698026.218933:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698026.218938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992448:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698026.218941:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.218947:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698026.218950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992576:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698026.218954:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698026.218962:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698026.218975:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531145993216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698026.218981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698026.218991:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698026.218995:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698026.218998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f5187388e00 x1874531145992576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698026.218998:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698026.219007:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5925 00000100:00080000:3.0:1787698026.219014:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698026.219019:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531145992576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698026.219026:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698026.219042:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531145993344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698026.219045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698026.219066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698026.219088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f5187389f80 x1874531145992704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698026.219110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531145992704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698026.219117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698026.219129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698026.219145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f518738b480 x1874531145992832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698026.219166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531145992832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698026.219173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145992832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698026.219196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145992960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698026.219215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f5187388000 x1874531145992960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698026.219236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531145992960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698026.219241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145992960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698026.219249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145993088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698026.219261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f518738a680 x1874531145993088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698026.219292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531145993088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698026.219296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145993088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698026.219303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145993216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698026.219314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f5187389180 x1874531145993216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698026.219326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531145993216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698026.219330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145993216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698026.219336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145993344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698026.219345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698026.219349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698026/real 1787698026] req@ffff8f518738b800 x1874531145993344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698026.219385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531145993344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698026.219390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698026.219392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698026.219394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145993344:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698027.114552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698028.138521:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698029.162518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698030.186547:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698031.210483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698031.338455:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338463:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698031.338466:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698031.338473:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338480:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338486:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698031.338515:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145993472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698031.338535:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338537:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698031.338538:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698031.338541:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338543:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338545:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698031.338553:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698031.338561:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531145993600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698031.338567:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338569:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698031.338571:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698031.338574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145993472:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698031.338575:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338580:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338583:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698031.338586:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698031.338597:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531145993728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698031.338606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338609:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698031.338611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698031.338614:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338618:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338622:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698031.338626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698031.338627:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698031.338641:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531145993856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698031.338647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698031.338651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698031.338653:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338657:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698031.338660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f518738aa00 x1874531145993472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698031.338661:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698031.338667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698031.338678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531145993984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698031.338683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145993472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698031.338688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698031.338694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698031.338694:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698031.338697:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698031.338698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.338705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145993472:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698031.338708:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338714:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338718:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698031.338724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993600:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698031.338724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698031.338743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531145994112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698031.338745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.338756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f518738a300 x1874531145993600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698031.338758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698031.338771:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698031.338775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531145993600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698031.338779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338786:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698031.338787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.338792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698031.338794:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698031.338798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993600:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698031.338801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698031.338803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698031.338815:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531145994240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698031.338821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698031.338827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698031.338832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f5187389500 x1874531145993728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698031.338834:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698031.338838:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338840:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338844:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698031.338848:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698031.338857:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531145994368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698031.338863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531145993728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698031.338874:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698031.338879:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698031.338881:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698031.338886:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5930 00000100:00080000:3.0:1787698031.338889:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698031.338894:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698031.338896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698031.338897:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698031.338902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.338907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698031.338907:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531145994496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698031.338918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698031.338937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.338946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f518738ad80 x1874531145993856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698031.338953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531145993856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698031.338957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.338959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.338961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698031.338964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698031.338972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.338977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f518738bb80 x1874531145993984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698031.338983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531145993984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698031.338987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.338989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.338991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145993984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698031.339000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698031.339009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.339014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f5187389c00 x1874531145994112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698031.339027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531145994112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698031.339032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.339034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.339036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698031.339039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698031.339048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.339055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f5187388a80 x1874531145994240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698031.339065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531145994240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698031.339072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.339075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.339078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698031.339084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698031.339101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.339108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f5187388380 x1874531145994368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698031.339115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531145994368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698031.339120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.339121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.339123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698031.339126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698031.339136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698031.339140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698031/real 1787698031] req@ffff8f518738aa00 x1874531145994496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698031.339146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531145994496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698031.339153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698031.339156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698031.339158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145994496:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698032.234521:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698033.258499:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698034.282522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698035.306482:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698036.330557:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698036.458485:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698036.458495:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698036.458498:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.458506:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00080000:2.0:1787698036.458517:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458522:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698036.458559:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145994624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698036.458589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698036.458600:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698036.458602:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.458607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00080000:2.0:1787698036.458611:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458615:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.458625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698036.458643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145994752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698036.458653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145994624:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698036.458661:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698036.458666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698036.458668:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.458672:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00080000:2.0:1787698036.458675:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458678:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.458682:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698036.458693:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145994880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698036.458699:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698036.458703:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698036.458706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.458709:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00080000:2.0:1787698036.458713:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458718:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.458724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698036.458729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698036.458739:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145995008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698036.458761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698036.458765:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698036.458766:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.458774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00100000:0.0:1787698036.458776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704dc00 x1874531145994624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698036.458780:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458791:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.458799:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698036.458823:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145995136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698036.458850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145994624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698036.458856:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698036.458875:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698036.458879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698036.458880:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698036.458888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698036.458893:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00100000:0.0:1787698036.458895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145994624:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698036.458905:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458913:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.458920:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698036.458927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145994752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698036.458948:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145995264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698036.458961:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698036.458964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698036.458966:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698036.458970:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.458981:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00100000:0.0:1787698036.458987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704ed80 x1874531145994752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698036.458988:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.458996:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.459006:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698036.459029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145994752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698036.459031:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145995392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698036.459098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698036.459100:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698036.459109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698036.459110:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698036.459116:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698036.459119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145994752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698036.459127:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00080000:2.0:1787698036.459133:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698036.459136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145994880:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698036.459144:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.459149:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698036.459162:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145995520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698036.459170:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698036.459174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698036.459180:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698036.459183:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698036.459188:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5936 00000100:00100000:0.0:1787698036.459192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704f480 x1874531145994880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698036.459193:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698036.459205:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698036.459215:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698036.459222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145994880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698036.459230:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145995648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698036.459292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145994880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698036.459311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145995008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698036.459332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698036.459340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704d500 x1874531145995008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698036.459347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145995008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698036.459399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145995008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698036.459434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145995136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698036.459456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698036.459468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704d880 x1874531145995136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698036.459481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145995136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698036.459487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145995136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698036.459506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698036.459521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698036.459528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704e300 x1874531145995264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698036.459535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145995264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698036.459540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698036.459548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698036.459558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698036.459564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704ea00 x1874531145995392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698036.459574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145995392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698036.459581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698036.459593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698036.459613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698036.459622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704dc00 x1874531145995520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698036.459629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145995520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698036.459634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698036.459641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698036.459652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698036.459658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698036/real 1787698036] req@ffff8f518704d180 x1874531145995648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698036.459664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145995648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698036.459669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698036.459670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698036.459672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145995648:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698037.354602:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698038.378475:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698039.402532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698040.426479:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698041.450530:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698041.578413:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578423:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698041.578426:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578436:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578445:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578451:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698041.578483:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145995776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698041.578507:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578511:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698041.578513:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578517:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578521:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578525:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698041.578534:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698041.578547:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145995904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698041.578551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145995776:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698041.578569:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578572:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698041.578574:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578579:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578583:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578587:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698041.578591:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698041.578601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145996032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698041.578609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698041.578616:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698041.578622:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578627:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578630:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578634:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698041.578639:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698041.578646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704fb80 x1874531145995776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698041.578651:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145996160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698041.578678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698041.578684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698041.578686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145995776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698041.578691:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578698:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698041.578701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698041.578705:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698041.578707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.578712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145995776:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787698041.578712:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698041.578727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145996288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698041.578735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145995904:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698041.578738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698041.578744:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578749:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5941 00000800:00000100:0.0:1787698041.578753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698041.578754:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578761:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698041.578765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704c700 x1874531145995904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698041.578766:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698041.578778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145995904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698041.578783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145996416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698041.578787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.578794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.578799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145995904:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698041.578800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698041.578807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145996032:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698041.578809:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578815:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578819:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578823:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698041.578828:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698041.578833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698041.578841:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145996544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698041.578849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704c380 x1874531145996032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698041.578877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698041.578882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145996032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698041.578882:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698041.578886:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578894:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5941 00000100:00080000:2.0:1787698041.578899:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578905:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698041.578913:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698041.578914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.578920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.578923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145996032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698041.578927:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145996672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698041.578931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145996160:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698041.578945:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698041.578949:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698041.578951:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698041.578956:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5941 00000800:00000100:0.0:1787698041.578958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698041.578962:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698041.578968:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698041.578970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704c000 x1874531145996160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698041.578975:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698041.578980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145996160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698041.578986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698041.578988:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145996800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698041.578990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.578996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145996160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698041.579031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145996288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698041.579045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698041.579052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704e680 x1874531145996288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698041.579059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145996288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698041.579063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.579065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.579067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145996288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698041.579071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698041.579080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698041.579085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704ce00 x1874531145996416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698041.579091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145996416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698041.579096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.579097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.579099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698041.579101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698041.579110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698041.579115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704f800 x1874531145996544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698041.579125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145996544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698041.579132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.579135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.579138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698041.579144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698041.579160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698041.579169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704df80 x1874531145996672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698041.579176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145996672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698041.579181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.579182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.579184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698041.579187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698041.579197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698041.579203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698041/real 1787698041] req@ffff8f518704fb80 x1874531145996800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698041.579209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145996800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698041.579213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698041.579215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698041.579216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145996800:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698042.474505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698043.498566:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698044.522516:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698045.546545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698046.570571:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698046.698509:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.698522:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698046.698527:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.698540:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.698554:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.698561:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698046.698602:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145996928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698046.698635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.698638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698046.698640:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.698645:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.698649:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.698653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698046.698664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698046.698678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145997056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698046.698688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.698691:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698046.698693:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.698697:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.698701:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.698705:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698046.698711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698046.698722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145996928:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698046.698725:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531145997184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698046.698745:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.698749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698046.698751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.698756:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.698761:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.698765:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698046.698771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698046.698784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531145997312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698046.698793:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698046.698795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698046.698802:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698046.698805:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.698811:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.698817:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.698822:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698046.698827:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698046.698841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704d180 x1874531145996928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698046.698845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531145997440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698046.698870:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.698877:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698046.698882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.698900:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.698907:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698046.698911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145996928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698046.698931:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698046.698950:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698046.698953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.698959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.698962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145996928:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698046.698969:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531145997568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698046.698970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698046.698993:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.698997:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698046.698999:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698046.699001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704dc00 x1874531145997056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698046.699022:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.699031:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.699039:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698046.699046:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698046.699051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145997056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698046.699064:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531145997696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698046.699071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997056:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698046.699094:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.699099:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698046.699101:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.699108:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00080000:2.0:1787698046.699112:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698046.699118:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997184:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698046.699127:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698046.699149:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531145997824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698046.699162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698046.699173:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698046.699179:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698046.699184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704ea00 x1874531145997184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698046.699185:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698046.699204:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5946 00000100:00100000:0.0:1787698046.699208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531145997184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698046.699215:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698046.699228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698046.699229:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698046.699234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997184:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698046.699242:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698046.699245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698046.699265:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531145997952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698046.699267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704e300 x1874531145997312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698046.699288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531145997312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.699297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698046.699302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698046.699313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704d880 x1874531145997440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698046.699327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531145997440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.699333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145997440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698046.699342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698046.699401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704d500 x1874531145997568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698046.699425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531145997568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.699437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698046.699446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698046.699464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704f480 x1874531145997696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698046.699483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531145997696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.699490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698046.699506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698046.699524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704ed80 x1874531145997824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698046.699546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531145997824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.699554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698046.699562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698046.699580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698046.699589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698046/real 1787698046] req@ffff8f518704dc00 x1874531145997952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698046.699599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531145997952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698046.699604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698046.699606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698046.699608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145997952:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698047.594578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698048.618574:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698049.642578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698050.666494:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698051.690581:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698051.818446:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818456:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698051.818460:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818482:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818488:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698051.818532:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531145998080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698051.818569:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818574:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698051.818576:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818581:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818585:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818589:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818599:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698051.818616:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531145998208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698051.818625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145998080:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698051.818651:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818654:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698051.818656:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818661:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818664:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818667:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818672:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698051.818681:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531145998336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698051.818688:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818690:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698051.818692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818694:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818696:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818698:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818702:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698051.818706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698051.818710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531145998464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698051.818717:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818719:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698051.818721:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818723:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818726:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818728:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818731:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698051.818736:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531145998592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698051.818741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698051.818751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818754:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818756:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818759:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698051.818764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704d180 x1874531145998080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698051.818772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531145998720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698051.818798:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698051.818809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531145998080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698051.818810:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698051.818814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818823:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:0.0:1787698051.818837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698051.818844:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698051.818845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.818851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145998080:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698051.818854:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818861:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698051.818874:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531145998848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698051.818879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998208:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698051.818893:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818896:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698051.818897:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818901:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818905:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698051.818907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698051.818910:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818915:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698051.818921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704fb80 x1874531145998208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698051.818926:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531145998976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698051.818940:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698051.818944:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698051.818945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531145998208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698051.818945:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698051.818951:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5951 00000100:00080000:2.0:1787698051.818955:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698051.818959:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698051.818965:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698051.818967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.818974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698051.818977:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531145999104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698051.818981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698051.818993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698051.819012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704df80 x1874531145998336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531145998336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698051.819045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698051.819055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704f800 x1874531145998464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531145998464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698051.819078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698051.819087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704ce00 x1874531145998592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531145998592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145998592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698051.819121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145998720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698051.819153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704e680 x1874531145998720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531145998720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145998720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698051.819193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145998848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698051.819205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704c000 x1874531145998848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531145998848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145998848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698051.819244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145998976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698051.819261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704c380 x1874531145998976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531145998976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145998976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698051.819288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145999104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698051.819298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698051.819303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698051/real 1787698051] req@ffff8f518704c700 x1874531145999104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698051.819309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531145999104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698051.819314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698051.819315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698051.819317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145999104:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698052.714516:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698053.738507:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698054.762551:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698055.786521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698056.810514:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698056.938458:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938466:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698056.938469:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938477:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938485:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938490:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698056.938519:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531145999232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938537:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938540:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698056.938541:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938544:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938546:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938548:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938555:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698056.938562:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531145999360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938569:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938571:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698056.938573:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938575:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938578:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938580:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938583:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698056.938588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145999232:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698056.938596:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531145999488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938607:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938610:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698056.938612:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938615:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938619:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938623:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698056.938639:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531145999616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938645:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938647:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698056.938649:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698056.938653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698056.938655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938659:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938661:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698056.938672:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531145999744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938678:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938680:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698056.938681:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938683:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938685:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938688:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938691:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698056.938698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183644e00 x1874531145999232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698056.938702:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531145999872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938727:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698056.938753:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938759:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00100000:0.0:1787698056.938763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531145999232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698056.938764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938772:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938781:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698056.938782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.938788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698056.938793:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146000000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698056.938795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531145999232:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698056.938805:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938808:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698056.938812:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938816:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00100000:0.0:1787698056.938819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999360:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698056.938820:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698056.938825:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938829:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698056.938838:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146000128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698056.938846:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698056.938851:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698056.938852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698056.938854:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698056.938860:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5956 00000100:00080000:1.0:1787698056.938864:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698056.938869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183646d80 x1874531145999360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698056.938870:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698056.938878:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698056.938894:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146000256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698056.938895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531145999360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.938918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.938922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.938925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698056.938949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698056.938967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.938977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183646a00 x1874531145999488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.938987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531145999488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698056.939011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698056.939025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.939034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183647b80 x1874531145999616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.939044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531145999616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698056.939061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698056.939075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.939082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183644380 x1874531145999744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.939092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531145999744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531145999744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698056.939110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145999872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698056.939124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.939132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183647100 x1874531145999872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.939142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531145999872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531145999872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698056.939160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146000000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698056.939173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.939181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183645180 x1874531146000000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.939190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146000000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146000000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698056.939208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146000128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698056.939221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.939230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183645c00 x1874531146000128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.939240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146000128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146000128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698056.939257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146000256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698056.939270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698056.939278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698056] req@ffff8f5183647480 x1874531146000256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698056.939288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146000256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698056.939295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698056.939297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698056.939300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146000256:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698057.834520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698058.858574:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698059.882567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698060.906514:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698061.930548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698062.058460:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.058474:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698062.058479:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.058492:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.058505:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.058513:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698062.058559:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146000384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.058792:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.058796:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698062.058798:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.058804:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.058808:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.058813:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.058825:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698062.058839:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146000512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.058849:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.058852:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698062.058854:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.058858:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.058861:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.058865:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.058870:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698062.058880:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146000640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.058889:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.058893:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698062.058895:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.059232:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.059238:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.059242:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.059248:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698062.059262:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146000768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.059272:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.059275:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698062.059277:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.059281:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.059284:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.059288:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.059292:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698062.059303:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146000896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.059311:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.059315:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698062.059316:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.059320:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.059323:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.059327:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.059333:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698062.059344:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146001024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.059394:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.059398:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698062.059399:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.059403:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.059407:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.059411:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.059416:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698062.059426:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146001152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.059434:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.059437:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698062.059438:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.059441:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.059444:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.059447:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.059450:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698062.059459:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146001280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698062.059467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698062.059470:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698062.059472:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698062.059475:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5961 00000100:00080000:1.0:1787698062.059479:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698062.059482:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698062.059486:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698062.059495:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146001408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698062.059744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146000384:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787698062.059807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.059849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183644700 x1874531146000384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.059874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146000384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.059884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.059888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.059892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146000384:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698062.059899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698062.059921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.059933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183644000 x1874531146000512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.059941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146000512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.059946:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.059948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.059950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698062.059954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698062.059967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.059973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183646680 x1874531146000640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.059980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146000640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.059988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.059990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.059992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698062.059995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698062.060004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.060009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183647800 x1874531146000768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.060015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146000768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.060020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.060022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.060023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698062.060026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698062.060035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.060040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183647480 x1874531146000896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.060046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146000896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.060050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.060052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.060054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146000896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698062.060057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698062.060067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.060072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183645c00 x1874531146001024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.060078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146001024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.060083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.060085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.060086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698062.060090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698062.060098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.060103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183645180 x1874531146001152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.060110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146001152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.060114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.060115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.060117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698062.060120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698062.060128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.060134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183647100 x1874531146001280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.060140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146001280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.060144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.060145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.060147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698062.060150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698062.060158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698062.060163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698062/real 1787698062] req@ffff8f5183644380 x1874531146001408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698062.060168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146001408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698062.060172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698062.060174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698062.060175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146001408:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698062.954587:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698063.978559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698065.002658:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698066.026623:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698067.050699:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698067.178499:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178508:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698067.178512:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178521:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178532:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178538:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698067.178586:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146001536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178652:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698067.178654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178659:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178663:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178667:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698067.178689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146001664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178698:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178701:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698067.178703:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178706:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178709:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178712:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178716:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698067.178725:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146001792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178733:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698067.178737:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178740:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178743:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178746:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698067.178758:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146001920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178765:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178768:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698067.178770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178773:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178775:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178778:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698067.178789:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146002048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178799:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698067.178801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178804:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178807:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178810:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178813:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698067.178821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146002176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698067.178838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178841:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178843:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178846:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698067.178857:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146002304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178865:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178867:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698067.178869:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178872:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178874:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178876:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698067.178886:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146002432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698067.178895:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698067.178897:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698067.178899:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698067.178901:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5966 00000100:00080000:3.0:1787698067.178904:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698067.178907:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698067.178911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698067.178918:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146002560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698067.178982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146001536:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787698067.179035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f518738b800 x1874531146001536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146001536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146001536:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698067.179111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146001664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698067.179130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f5187389180 x1874531146001664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146001664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146001664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698067.179171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146001792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698067.179188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f518738a680 x1874531146001792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146001792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146001792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698067.179231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146001920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698067.179246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f5187388000 x1874531146001920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146001920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146001920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698067.179282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146002048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698067.179297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f518738b480 x1874531146002048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146002048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146002048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698067.179333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698067.179348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f5187389f80 x1874531146002176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146002176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698067.179529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698067.179553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f5187388e00 x1874531146002304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146002304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698067.179602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698067.179617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f518738b100 x1874531146002432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146002432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698067.179716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698067.179734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698067.179742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698067/real 1787698067] req@ffff8f518738aa00 x1874531146002560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698067.179750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146002560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698067.179756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698067.179759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698067.179761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146002560:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698068.074617:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698069.098587:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698070.122554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698071.147055:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698072.170575:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698072.298411:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698072.298418:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698072.298421:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.298429:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298466:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.298473:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698072.298518:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146002688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698072.298544:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698072.298548:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698072.298550:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.298554:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.298563:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698072.298572:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698072.298586:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146002816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698072.298590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146002688:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698072.298611:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698072.298616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698072.298618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.298621:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298624:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.298627:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698072.298631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698072.298640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146002944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698072.298647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698072.298649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698072.298650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698072.298653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698072.298671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298674:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.298676:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698072.298688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f5187388380 x1874531146002688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.298712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146002688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.298721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.298727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.298731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146002688:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698072.298751:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698072.298756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146002816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698072.298768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146003072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698072.298779:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698072.298784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698072.298785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698072.298789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.298794:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298798:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698072.298803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f5187388a80 x1874531146002816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698072.298803:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698072.298809:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698072.298821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146003200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698072.298826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146002816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698072.298831:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698072.298841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.298846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698072.298849:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787698072.298852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146002816:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698072.298856:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.298862:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298866:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.298869:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698072.298874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146002944:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698072.298875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698072.298892:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146003328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698072.298902:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698072.298905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698072.298906:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698072.298909:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.298913:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.298917:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.298920:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698072.298924:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698072.298925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f5187389c00 x1874531146002944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698072.298937:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146003456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698072.298941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146002944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698072.298956:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698072.298963:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698072.298966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698072.298971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698072.298973:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:0.0:1787698072.298977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698072.298979:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698072.298984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146002944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698072.298985:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698072.298990:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698072.299000:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146003584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698072.299006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146003072:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698072.299009:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698072.299014:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698072.299015:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698072.299018:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5971 00000100:00080000:3.0:1787698072.299020:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698072.299022:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698072.299025:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698072.299032:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146003712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698072.299037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698072.299055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f518738bb80 x1874531146003072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.299067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146003072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.299075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.299078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.299081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146003072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698072.299087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146003200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698072.299104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698072.299114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f518738ad80 x1874531146003200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.299124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146003200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.299131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.299134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.299137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146003200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698072.299142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698072.299159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698072.299170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f5187389500 x1874531146003328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.299180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146003328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.299187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.299190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.299193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698072.299198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698072.299216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698072.299226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f518738a300 x1874531146003456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.299234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146003456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.299238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.299240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.299241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698072.299252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698072.299266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698072.299276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f5187388380 x1874531146003584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.299286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146003584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.299294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.299297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.299300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698072.299305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698072.299322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698072.299332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698072/real 1787698072] req@ffff8f518738aa00 x1874531146003712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698072.299342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146003712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698072.299350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698072.299401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698072.299404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146003712:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698073.194609:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698074.218554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698075.242542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698076.266562:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698077.290664:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698077.418451:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418461:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698077.418473:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418483:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00080000:3.0:1787698077.418494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418501:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698077.418537:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146003840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698077.418565:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698077.418569:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418576:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00080000:3.0:1787698077.418579:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418582:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418589:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698077.418599:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146003968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698077.418606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418608:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698077.418610:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418613:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00100000:0.0:1787698077.418618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146003840:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698077.418619:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698077.418647:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146004096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698077.418657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698077.418662:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418665:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00080000:3.0:1787698077.418670:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418674:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698077.418679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418680:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698077.418695:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146004224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698077.418703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418705:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698077.418707:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418710:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00080000:3.0:1787698077.418713:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418716:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698077.418721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f518738b100 x1874531146003840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698077.418729:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146004352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698077.418748:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698077.418757:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418760:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00080000:3.0:1787698077.418763:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418765:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418768:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698077.418771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146003840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698077.418776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146004480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698077.418798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698077.418801:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698077.418806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698077.418807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418812:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00100000:0.0:1787698077.418816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146003840:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698077.418816:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418821:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698077.418832:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146004608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698077.418837:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418842:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698077.418843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146003968:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698077.418845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418853:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00080000:3.0:1787698077.418857:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418860:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698077.418866:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698077.418872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698077.418879:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146004736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698077.418888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f5187388e00 x1874531146003968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698077.418893:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698077.418900:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698077.418902:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698077.418909:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5977 00000100:00100000:0.0:1787698077.418914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146003968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698077.418914:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698077.418922:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698077.418930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698077.418931:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698077.418937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.418941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146003968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698077.418947:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146004864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698077.418966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146004096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698077.418986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.418996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f5187389f80 x1874531146004096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146004096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146004096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698077.419030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146004224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698077.419044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.419063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f518738b480 x1874531146004224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146004224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146004224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698077.419093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146004352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698077.419108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.419117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f5187388000 x1874531146004352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146004352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146004352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698077.419149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698077.419167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.419177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f518738a680 x1874531146004480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146004480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698077.419208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698077.419225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.419235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f5187389180 x1874531146004608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146004608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698077.419266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698077.419282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.419291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f518738b800 x1874531146004736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146004736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698077.419319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698077.419333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698077.419341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698077/real 1787698077] req@ffff8f518738aa00 x1874531146004864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698077.419351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146004864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698077.419386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698077.419402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698077.419405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146004864:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698078.314560:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698079.338613:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698080.362531:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698081.386525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698082.410531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698082.538456:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538463:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698082.538466:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538473:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538481:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538486:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698082.538519:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146004992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538543:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538547:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698082.538549:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538553:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538557:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538561:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538571:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698082.538585:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146005120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538601:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538606:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698082.538609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146004992:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698082.538609:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538614:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538619:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538623:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538626:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698082.538647:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146005248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538661:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698082.538662:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538666:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538670:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538674:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538679:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698082.538687:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146005376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698082.538698:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538701:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538704:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698082.538710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005120:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698082.538710:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538713:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698082.538725:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146005504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698082.538741:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698082.538748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698082.538748:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538761:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698082.538780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698082.538782:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146005632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538799:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698082.538801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538805:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538809:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698082.538813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005504:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698082.538815:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538824:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698082.538837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146005760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538853:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698082.538859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146005632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698082.538860:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698082.538868:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538873:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538877:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538882:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698082.538888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146005760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698082.538895:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146005888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698082.538906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698082.538909:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698082.538911:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698082.538916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146005888:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698082.538916:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5982 00000100:00080000:3.0:1787698082.538920:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698082.538926:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698082.538931:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698082.538944:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146006016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698082.538979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146006016:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698083.434501:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698084.458529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698085.482484:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698085.610587:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787698085.610627:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698085.610643:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610692:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610705:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610713:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610731:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610738:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610745:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610752:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698085.610759:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698085.610849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f518738b800 x1874531146006016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.610878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146006016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.610890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.610894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.610897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146006016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.610904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f518738aa00 x1874531146005888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.610925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146005888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.610932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.610936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.610938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146005888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.610944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f5187388a80 x1874531146005760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.610953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146005760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.610959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.610962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.610964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146005760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698085.610970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f518738bb80 x1874531146005504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.610980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146005504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.610986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.610990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.610992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.610998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f5187389c00 x1874531146005632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.611007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146005632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.611015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.611019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.611021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146005632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698085.611027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f518738a300 x1874531146005120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.611038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146005120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.611045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.611049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.611052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698085.611058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f5187389500 x1874531146005248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.611068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146005248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.611075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.611079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.611082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698085.611088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f518738ad80 x1874531146005376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.611098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146005376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.611105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.611108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.611111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146005376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698085.611116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698082/real 1787698085] req@ffff8f5187388380 x1874531146004992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698085.611126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146004992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698085.611132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698085.611135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698085.611138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146004992:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787698086.506538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698087.530481:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698087.658449:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658456:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698087.658459:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658467:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658477:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658482:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698087.658508:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146006144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698087.658533:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658537:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698087.658538:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658542:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658545:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658549:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658557:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698087.658571:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146006272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698087.658580:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658584:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698087.658587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146006144:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698087.658587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658595:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658599:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658607:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698087.658621:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146006400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698087.658630:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698087.658634:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658636:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658640:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658642:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698087.658651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658652:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698087.658667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146006528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698087.658677:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698087.658682:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658686:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00100000:0.0:1787698087.658691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f5187389180 x1874531146006144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698087.658692:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658700:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698087.658720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146006656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698087.658724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146006144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698087.658743:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698087.658747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698087.658748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698087.658752:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698087.658756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698087.658761:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00100000:0.0:1787698087.658763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146006144:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698087.658767:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658774:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658779:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698087.658791:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146006784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698087.658794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006272:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698087.658810:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658813:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698087.658815:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658820:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658825:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698087.658829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698087.658833:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658839:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698087.658848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f518738a680 x1874531146006272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698087.658851:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146006912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698087.658865:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658872:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698087.658874:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658887:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658892:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698087.658896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146006272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698087.658900:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658909:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698087.658912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.658918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698087.658924:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146007040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698087.658925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698087.658937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006400:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698087.658940:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698087.658946:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698087.658949:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698087.658954:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5987 00000100:00080000:3.0:1787698087.658958:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698087.658963:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698087.658969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698087.658969:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698087.658985:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146007168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698087.658988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f5187388000 x1874531146006400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146006400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698087.659035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698087.659055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698087.659065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f518738b480 x1874531146006528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146006528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698087.659097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698087.659114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698087.659125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f5187389f80 x1874531146006656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146006656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146006656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698087.659163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146006784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698087.659181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698087.659191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f5187388e00 x1874531146006784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146006784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146006784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698087.659238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146006912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698087.659256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698087.659266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f518738b100 x1874531146006912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146006912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146006912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698087.659296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146007040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698087.659314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698087.659324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f5187389180 x1874531146007040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146007040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146007040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698087.659388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146007168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698087.659409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698087.659420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698087/real 1787698087] req@ffff8f5187388380 x1874531146007168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698087.659431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146007168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698087.659438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698087.659441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698087.659444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146007168:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698088.554548:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698089.578478:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698090.602516:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698091.626562:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698092.650551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698092.778455:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778465:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698092.778469:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778479:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778490:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778497:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698092.778535:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146007296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698092.778561:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778564:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698092.778565:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778568:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778571:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778573:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.778582:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698092.778596:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146007424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698092.778606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698092.778613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778617:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778622:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.778631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698092.778644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146007552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698092.778652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146007296:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698092.778659:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698092.778665:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778675:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778680:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.778685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698092.778705:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146007680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698092.778714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778717:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698092.778720:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778726:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5992 00000800:00000100:1.0:1787698092.778730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698092.778740:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778747:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.778753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698092.778768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146007808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698092.778778:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778781:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698092.778786:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698092.778788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace90700 x1874531146007296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698092.778795:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778803:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778812:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.778821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698092.778830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146007296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698092.778838:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146007936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698092.778856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698092.778858:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778863:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698092.778865:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778869:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778873:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.778881:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698092.778882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787698092.778889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698092.778893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146007296:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698092.778907:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146008064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698092.778908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007424:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698092.778932:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.778935:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698092.778936:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.778943:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.778948:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698092.778954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698092.778955:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.778962:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698092.778986:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146008192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698092.778987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace90000 x1874531146007424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698092.779010:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698092.779018:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698092.779023:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698092.779031:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5992 00000100:00080000:0.0:1787698092.779034:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698092.779037:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698092.779041:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698092.779050:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146008320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698092.779054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146007424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698092.779107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698092.779123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace93800 x1874531146007552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146007552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698092.779182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698092.779201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace93b80 x1874531146007680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146007680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698092.779226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698092.779234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace90380 x1874531146007808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146007808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146007808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698092.779257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146007936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698092.779266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace91180 x1874531146007936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146007936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146007936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698092.779288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146008064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698092.779297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace92680 x1874531146008064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146008064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146008064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698092.779319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146008192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698092.779327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace91c00 x1874531146008192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146008192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146008192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698092.779349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146008320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698092.779406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698092.779417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698092/real 1787698092] req@ffff8f51ace93100 x1874531146008320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698092.779427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146008320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698092.779432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698092.779434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698092.779436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146008320:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698093.674519:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698094.698500:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698095.722530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698096.746573:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698097.770595:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698097.898412:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898420:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698097.898423:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898430:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00080000:0.0:1787698097.898438:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898444:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698097.898477:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146008448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698097.898504:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898507:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698097.898509:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898512:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00080000:0.0:1787698097.898515:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898518:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698097.898526:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698097.898536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146008448:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698097.898537:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146008576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698097.898549:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898552:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698097.898553:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898555:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00080000:0.0:1787698097.898557:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898559:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698097.898562:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698097.898569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146008704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698097.898575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898577:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698097.898578:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898580:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00080000:0.0:1787698097.898591:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898593:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698097.898595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698097.898599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698097.898603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146008832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698097.898610:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898612:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698097.898614:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898616:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00080000:0.0:1787698097.898618:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898619:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698097.898622:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698097.898628:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146008960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698097.898634:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898636:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698097.898638:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698097.898640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace90e00 x1874531146008448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.898642:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00080000:0.0:1787698097.898655:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698097.898659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146008448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.898660:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698097.898667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698097.898671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.898675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.898679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146008448:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698097.898682:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146009088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698097.898693:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698097.898700:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898705:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00100000:1.0:1787698097.898705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008576:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698097.898710:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898715:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698097.898720:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698097.898726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698097.898731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146009216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698097.898737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace91880 x1874531146008576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.898741:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698097.898747:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698097.898750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898754:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00100000:1.0:1787698097.898756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146008576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.898758:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698097.898764:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698097.898766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698097.898767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698097.898771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.898774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698097.898780:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146009344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698097.898781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698097.898798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698097.898805:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698097.898808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace91500 x1874531146008704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.898811:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698097.898815:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698097.898823:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 5997 00000100:00100000:1.0:1787698097.898824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146008704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.898828:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698097.898835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.898838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698097.898840:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698097.898840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008704:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698097.898847:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698097.898853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698097.898858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146009472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698097.898869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698097.898877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace92300 x1874531146008832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698097.898884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146008832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698097.898889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.898891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.898892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698097.898900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698097.898910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698097.898915:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace93480 x1874531146008960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698097.898921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146008960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698097.898925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.898927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.898929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146008960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698097.898932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698097.898940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698097.898944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace92d80 x1874531146009088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698097.898951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146009088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698097.898956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.898957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.898959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698097.898968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698097.898977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698097.898982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace90e00 x1874531146009216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698097.898988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146009216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698097.898992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.898994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.898996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698097.898998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698097.899006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698097.899010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace93100 x1874531146009344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698097.899016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146009344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698097.899024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.899026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.899029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698097.899034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698097.899075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698097.899086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff8f51ace91c00 x1874531146009472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698097.899096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146009472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698097.899103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698097.899106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698097.899109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146009472:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698098.794505:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698099.818590:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698100.842570:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698101.866602:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698102.890514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698103.018418:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018425:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698103.018429:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018436:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018444:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018449:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698103.018478:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146009600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698103.018504:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018506:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698103.018510:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018513:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018516:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018518:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018525:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698103.018533:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146009728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698103.018540:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018543:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698103.018544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018546:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018553:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018555:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018560:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698103.018567:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146009856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698103.018576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018579:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698103.018582:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018586:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00100000:1.0:1787698103.018586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146009600:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698103.018593:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018612:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018618:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698103.018637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146009984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698103.018651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018655:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698103.018658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018663:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018676:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018681:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018686:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698103.018691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698103.018702:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146010112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698103.018710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018713:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698103.018715:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018720:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018725:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018728:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018733:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698103.018749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146010240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698103.018758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698103.018758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace92680 x1874531146009600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698103.018773:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698103.018777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018787:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018798:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018801:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698103.018821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146010368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698103.018828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146009600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698103.018833:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018841:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698103.018844:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018855:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00080000:0.0:1787698103.018863:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018874:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698103.018879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698103.018884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146010496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698103.018891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.018908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146009600:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698103.018909:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698103.018927:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698103.018930:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698103.018935:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6002 00000100:00100000:1.0:1787698103.018938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146009728:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698103.018940:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698103.018959:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698103.018964:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698103.018979:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146010624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698103.019011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace91180 x1874531146009728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146009728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146009728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698103.019095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146009856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698103.019116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace90380 x1874531146009856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146009856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146009856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698103.019160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146009984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698103.019178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace93b80 x1874531146009984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146009984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146009984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698103.019222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146010112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698103.019239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace93800 x1874531146010112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146010112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146010112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698103.019282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698103.019300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace90000 x1874531146010240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146010240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698103.019345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698103.019412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace90700 x1874531146010368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146010368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698103.019458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698103.019477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace91c00 x1874531146010496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146010496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698103.019519:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698103.019536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698103.019546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698103/real 1787698103] req@ffff8f51ace93100 x1874531146010624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698103.019557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146010624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698103.019566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698103.019569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698103.019572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146010624:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698103.914538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698104.938506:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698105.962543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698106.986522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698108.010518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698108.138480:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138489:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698108.138493:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.138503:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138512:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698108.138518:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698108.138558:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146010752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698108.138583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698108.138587:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.138591:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138594:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698108.138597:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698108.138605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698108.138615:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146010880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698108.138624:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138627:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698108.138628:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.138632:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698108.138638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146010752:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698108.138641:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698108.138649:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698108.138659:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146011008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698108.138667:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138670:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698108.138672:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.138675:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138692:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698108.138696:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698108.138700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698108.138705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698108.138712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146011136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698108.138721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138724:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698108.138726:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.138729:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138732:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698108.138736:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698108.138740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698108.138749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace90e00 x1874531146010752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698108.138752:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146011264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698108.138783:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698108.138788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146010752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698108.138789:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698108.138794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.138804:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138808:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698108.138815:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698108.138816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698108.138822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698108.138824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.138831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146010752:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698108.138838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146010880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698108.138855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146011392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698108.138888:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138892:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698108.138893:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698108.138896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698108.138900:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.138906:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698108.138910:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698108.138915:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace92d80 x1874531146010880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698108.138916:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698108.138934:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146011520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698108.138947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146010880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698108.138949:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.138959:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698108.138963:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698108.138967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698108.138970:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:1.0:1787698108.138975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698108.138977:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698108.138982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146010880:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698108.138987:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698108.138993:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698108.139005:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146011648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698108.139013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146011008:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698108.139017:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698108.139023:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698108.139025:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698108.139030:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6007 00000100:00080000:0.0:1787698108.139035:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698108.139040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698108.139042:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698108.139048:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698108.139057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace93480 x1874531146011008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698108.139065:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146011776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698108.139076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146011008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146011008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698108.139099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146011136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698108.139117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698108.139127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace92300 x1874531146011136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698108.139142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146011136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146011136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698108.139160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146011264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698108.139177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698108.139186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace91500 x1874531146011264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698108.139196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146011264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146011264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698108.139214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698108.139228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698108.139236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace91880 x1874531146011392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698108.139246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146011392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698108.139268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698108.139284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698108.139293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace93100 x1874531146011520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698108.139303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146011520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698108.139330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698108.139347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698108.139374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace91c00 x1874531146011648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698108.139382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146011648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698108.139394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698108.139405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698108.139411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698108/real 1787698108] req@ffff8f51ace90700 x1874531146011776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698108.139418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146011776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698108.139422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698108.139424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698108.139426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146011776:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698109.034508:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698110.058502:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698111.082527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698112.106500:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698113.130620:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698113.258474:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258489:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698113.258494:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258505:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258518:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258524:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698113.258565:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146011904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258594:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258597:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698113.258598:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258601:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258603:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258607:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258615:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698113.258625:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146012032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258631:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698113.258635:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258637:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258639:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258641:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698113.258652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146011904:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698113.258654:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146012160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258666:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698113.258669:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258671:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258673:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258674:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258677:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698113.258685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146012288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698113.258694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258696:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258701:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698113.258709:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146012416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698113.258717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258720:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258725:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698113.258729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698113.258738:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146012544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258746:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258749:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698113.258751:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258754:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258759:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698113.258762:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258768:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698113.258791:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146012672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698113.258793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f518738ad80 x1874531146011904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698113.258824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698113.258830:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698113.258832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146011904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698113.258839:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:1.0:1787698113.258845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698113.258846:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698113.258851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698113.258855:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698113.258856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146011904:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698113.258863:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698113.258865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698113.258876:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146012800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698113.258884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698113.258886:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698113.258888:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698113.258890:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6012 00000100:00080000:3.0:1787698113.258894:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698113.258895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698113.258899:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698113.258903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698113.258908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f5187389500 x1874531146012032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698113.258913:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146012928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698113.258921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146012032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.258929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.258932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.258934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698113.258956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698113.258966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.258972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f518738a300 x1874531146012160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.258978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146012160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.258983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.258985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.258986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698113.258989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698113.258998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.259003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f5187389c00 x1874531146012288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.259014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146012288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.259019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.259020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.259022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698113.259025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698113.259053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.259064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f518738bb80 x1874531146012416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.259074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146012416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.259082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.259085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.259088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146012416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698113.259093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698113.259111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.259121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f5187388a80 x1874531146012544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.259132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146012544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.259145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.259148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.259151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698113.259156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698113.259174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.259184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f518738aa00 x1874531146012672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.259196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146012672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.259203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.259206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.259208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698113.259212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698113.259225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.259232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f518738b800 x1874531146012800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.259239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146012800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.259243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.259245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.259248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698113.259251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698113.259260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698113.259265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698113/real 1787698113] req@ffff8f5187388380 x1874531146012928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698113.259271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146012928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698113.259276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698113.259277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698113.259279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146012928:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698114.154528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698115.178533:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698116.202551:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698117.226547:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698118.250466:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698118.378463:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378470:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698118.378474:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378482:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378490:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378495:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698118.378526:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146013056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378546:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378549:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698118.378550:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378553:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378555:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378557:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378563:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698118.378570:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146013184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698118.378579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378581:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378583:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378584:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378587:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698118.378594:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146013312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378600:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378602:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698118.378603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378605:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378608:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378611:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378613:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698118.378619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146013440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378624:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378625:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698118.378627:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378630:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378632:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378634:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378639:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698118.378647:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146013568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698118.378651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146013056:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698118.378657:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378663:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698118.378665:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378668:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378674:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378677:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378680:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698118.378685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146013696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378707:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698118.378708:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378710:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378712:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378715:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378717:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698118.378729:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146013824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698118.378737:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378739:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378740:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378742:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378746:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698118.378751:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146013952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698118.378760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698118.378762:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698118.378763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698118.378765:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6018 00000100:00080000:3.0:1787698118.378767:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698118.378770:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698118.378771:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698118.378776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146014080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698118.378790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.378848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f5187389180 x1874531146013056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.378869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146013056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.378881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.378885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.378889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146013056:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698118.378898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698118.378921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.378932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f518738b100 x1874531146013184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.378944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146013184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.378951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.378955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.378957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698118.378963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698118.378982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.378999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f5187388e00 x1874531146013312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146013312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698118.379033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698118.379051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.379060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f5187389f80 x1874531146013440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146013440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698118.379121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698118.379138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.379148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f518738b480 x1874531146013568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146013568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146013568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698118.379179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146013696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698118.379194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.379203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f5187388000 x1874531146013696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146013696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146013696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698118.379239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146013824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698118.379253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.379262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f518738a680 x1874531146013824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146013824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146013824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698118.379291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146013952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698118.379306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.379314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f5187388380 x1874531146013952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146013952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146013952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698118.379343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146014080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698118.379396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698118.379406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698118/real 1787698118] req@ffff8f518738b800 x1874531146014080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698118.379417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146014080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698118.379424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698118.379427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698118.379430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146014080:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698119.274583:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698120.298592:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698121.322594:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698122.346529:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698123.370545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698123.498492:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498502:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698123.498506:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698123.498516:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:2.0:1787698123.498527:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.498534:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698123.498571:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146014208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698123.498597:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698123.498615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698123.498620:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:2.0:1787698123.498623:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.498628:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698123.498638:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698123.498645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146014208:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698123.498661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146014336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698123.498674:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498677:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698123.498679:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698123.498682:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:2.0:1787698123.498686:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.498690:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698123.498694:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698123.498695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698123.498709:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146014464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698123.498719:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498722:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698123.498724:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698123.498728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704c700 x1874531146014208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698123.498729:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:2.0:1787698123.498736:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.498743:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698123.498748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146014208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.498757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698123.498760:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698123.498761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.498766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146014208:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698123.498776:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146014592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698123.498785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014336:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698123.498789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498793:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698123.498795:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698123.498800:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6023 00000800:00000100:1.0:1787698123.498804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698123.498805:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.498811:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698123.498815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704c380 x1874531146014336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698123.498817:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698123.498829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146014336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698123.498834:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146014720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698123.498837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698123.498844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698123.498849:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698123.498851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014336:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698123.498855:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698123.498859:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698123.498862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698123.498874:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:2.0:1787698123.498879:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698123.498884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698123.498886:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698123.498893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698123.498899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704c000 x1874531146014464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698123.498908:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146014848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698123.498916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146014464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698123.498922:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498930:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698123.498933:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698123.498934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698123.498938:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:1.0:1787698123.498940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.498945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698123.498945:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.498953:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698123.498958:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698123.498961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698123.498971:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146014976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698123.498980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698123.498982:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698123.498986:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698123.498988:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698123.498993:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00100000:1.0:1787698123.498995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704e680 x1874531146014592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698123.498998:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.499005:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698123.499011:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698123.499025:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146015104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698123.499029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146014592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.499049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698123.499049:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698123.499053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.499056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014592:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698123.499062:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698123.499064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014720:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698123.499066:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698123.499072:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6023 00000100:00080000:2.0:1787698123.499076:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698123.499080:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698123.499084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698123.499087:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698123.499094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704ce00 x1874531146014720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698123.499101:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146015232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698123.499104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146014720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.499122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698123.499126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.499129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146014720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698123.499133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146014848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698123.499144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698123.499150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704f800 x1874531146014848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698123.499156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146014848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.499165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698123.499167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.499169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146014848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698123.499180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146014976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698123.499196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698123.499205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704df80 x1874531146014976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698123.499223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146014976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.499228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698123.499230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.499232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146014976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698123.499235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146015104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698123.499247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698123.499253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704fb80 x1874531146015104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698123.499262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146015104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.499268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698123.499271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.499275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146015104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698123.499280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146015232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698123.499306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698123.499314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698123/real 1787698123] req@ffff8f518704d180 x1874531146015232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698123.499321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146015232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698123.499325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698123.499327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698123.499329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146015232:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698124.394617:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698125.418586:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698126.442493:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698127.466529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698128.490619:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698128.618469:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.618483:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698128.618493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698128.618507:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.618531:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698128.618539:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698128.618589:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146015360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698128.618625:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.618629:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698128.618631:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698128.618637:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.618640:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698128.618646:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698128.618658:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698128.618675:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146015488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698128.618676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146015360:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698128.618712:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.618715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698128.618717:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698128.618722:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.618727:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698128.618733:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698128.618739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698128.618752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146015616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698128.618754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698128.618768:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.618772:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698128.618774:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698128.618780:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.618785:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698128.618790:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698128.618795:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698128.618828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146015744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698128.618829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704dc00 x1874531146015360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698128.618896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.618901:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698128.618906:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698128.618908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146015360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698128.618917:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.618929:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698128.618934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698128.618938:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698128.618944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698128.618949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698128.618952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146015360:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698128.618969:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146015872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698128.618980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015488:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698128.618983:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.618989:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698128.618991:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698128.618998:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.619002:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698128.619008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698128.619013:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698128.619021:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698128.619022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704ed80 x1874531146015488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698128.619049:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146016000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698128.619050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146015488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698128.619093:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698128.619097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698128.619098:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698128.619104:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698128.619108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698128.619114:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00100000:1.0:1787698128.619119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015488:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698128.619126:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698128.619133:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015616:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698128.619143:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698128.619164:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146016128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698128.619177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704f480 x1874531146015616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698128.619195:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698128.619202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146015616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698128.619202:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698128.619213:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698128.619232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698128.619233:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:1.0:1787698128.619243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698128.619245:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698128.619249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015616:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698128.619255:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015744:192.168.201.125@tcp:8:kworker.0 00000008:00100000:2.0:1787698128.619274:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698128.619285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698128.619294:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146016256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698128.619296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704d500 x1874531146015744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698128.619385:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698128.619390:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698128.619393:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698128.619398:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6028 00000100:00080000:2.0:1787698128.619407:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698128.619419:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698128.619427:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698128.619428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146015744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698128.619441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698128.619450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698128.619451:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146016384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698128.619467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698128.619492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698128.619531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704d880 x1874531146015872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698128.619555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146015872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698128.619564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698128.619567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698128.619569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146015872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698128.619574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698128.619589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704e300 x1874531146016000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698128.619603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146016000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698128.619612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698128.619614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698128.619616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698128.619619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698128.619628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704ea00 x1874531146016128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698128.619640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146016128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698128.619644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698128.619645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698128.619647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698128.619650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698128.619659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704dc00 x1874531146016256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698128.619671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146016256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698128.619675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698128.619676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698128.619678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698128.619682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698128.619691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698128.619695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698128/real 1787698128] req@ffff8f518704d180 x1874531146016384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698128.619701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146016384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698128.619706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698128.619707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698128.619709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146016384:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698129.514536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698130.538571:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698131.562524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698132.586486:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698133.610512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698133.738473:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738480:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698133.738484:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738493:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738515:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738522:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698133.738558:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146016512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698133.738579:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698133.738583:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738586:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738589:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738591:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738599:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698133.738606:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146016640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698133.738612:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738614:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698133.738615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738617:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738619:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738621:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738623:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698133.738632:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146016768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698133.738633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146016512:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698133.738641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698133.738646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738648:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738649:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738651:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698133.738660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146016896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698133.738669:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698133.738673:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738675:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738683:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738685:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738687:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698133.738692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698133.738696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146017024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698133.738702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738705:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698133.738706:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738709:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738710:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738712:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738715:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698133.738724:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146017152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698133.738729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183647b80 x1874531146016512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698133.738734:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698133.738740:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698133.738744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738747:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738750:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698133.738753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146016512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698133.738756:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698133.738767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698133.738772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146017280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698133.738774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.738780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146016512:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698133.738787:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698133.738791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146016640:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698133.738791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698133.738795:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738800:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738803:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698133.738807:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698133.738811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738813:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698133.738821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183646a00 x1874531146016640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698133.738828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146017408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698133.738833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146016640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698133.738853:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698133.738854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.738861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698133.738861:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698133.738865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698133.738869:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6033 00000100:00080000:1.0:1787698133.738871:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698133.738874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146016640:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698133.738874:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698133.738882:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698133.738891:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146017536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698133.738899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146016768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698133.738917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.738927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183646d80 x1874531146016768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.738934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146016768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.738939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.738941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.738943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146016768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698133.738946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146016896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698133.738955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.738960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183644e00 x1874531146016896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.738966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146016896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.738970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.738972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.738973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146016896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698133.738976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146017024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698133.738984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.738989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183644380 x1874531146017024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.738995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146017024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.739003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.739005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.739007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146017024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698133.739009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698133.739018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.739023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183647100 x1874531146017152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.739028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146017152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.739033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.739035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.739037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698133.739039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698133.739047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.739052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183645180 x1874531146017280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.739058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146017280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.739062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.739064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.739066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698133.739069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698133.739077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.739082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183645c00 x1874531146017408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.739087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146017408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.739091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.739093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.739094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698133.739105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698133.739113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698133.739118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698133/real 1787698133] req@ffff8f5183647480 x1874531146017536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698133.739123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146017536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698133.739127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698133.739129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698133.739131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146017536:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698134.634524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698135.658481:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698136.682600:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698137.706576:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698138.730579:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698138.858512:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858524:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698138.858528:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.858541:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858554:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698138.858571:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698138.858624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146017664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698138.858653:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698138.858659:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.858664:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858668:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698138.858674:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.858685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698138.858685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146017664:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698138.858701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146017792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698138.858713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858717:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698138.858719:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.858723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858727:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698138.858731:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.858737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698138.858748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146017920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698138.858757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698138.858760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698138.858763:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698138.858766:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.858771:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858775:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698138.858780:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.858785:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698138.858796:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146018048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698138.858806:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698138.858811:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698138.858816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace90000 x1874531146017664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.858819:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858825:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698138.858832:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.858838:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698138.858842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146017664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.858851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698138.858853:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146018176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698138.858856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.858863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146017664:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698138.858874:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858878:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698138.858881:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.858886:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858890:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698138.858896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146017792:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698138.858897:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.858905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698138.858920:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146018304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698138.858928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698138.858937:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858941:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698138.858942:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698138.858944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace93800 x1874531146017792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.858949:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.858955:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698138.858959:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.858963:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698138.858974:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146018432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698138.858981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146017792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.858982:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.858988:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698138.858991:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.858999:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:1.0:1787698138.859003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698138.859004:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698138.859012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698138.859013:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.859021:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698138.859021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146017792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698138.859030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146017920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698138.859039:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146018560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698138.859049:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.859054:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698138.859054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698138.859057:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.859062:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6038 00000100:00080000:0.0:1787698138.859065:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698138.859068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace93b80 x1874531146017920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.859071:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698138.859079:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698138.859084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146017920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698138.859091:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146018688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698138.859095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146017920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698138.859111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146018048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698138.859125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698138.859132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace90380 x1874531146018048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.859139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146018048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.859147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146018048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698138.859151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146018176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698138.859160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698138.859165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace91180 x1874531146018176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.859175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146018176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.859181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146018176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698138.859203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698138.859214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698138.859220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace92680 x1874531146018304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.859227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146018304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.859235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698138.859239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698138.859252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698138.859257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace90000 x1874531146018432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.859264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146018432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.859277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698138.859286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698138.859303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698138.859314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace90700 x1874531146018560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.859325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146018560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.859336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698138.859390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698138.859405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698138.859412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff8f51ace91c00 x1874531146018688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.859420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146018688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.859425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.859426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698138.859429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146018688:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698139.754581:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698140.778537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698141.802498:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698142.826486:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698143.850499:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698143.978453:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978460:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698143.978463:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978470:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978479:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698143.978484:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698143.978518:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146018816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698143.978538:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978540:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698143.978542:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978545:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978547:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698143.978549:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978556:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698143.978564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146018944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698143.978570:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978572:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698143.978573:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978575:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978577:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698143.978580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146018816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698143.978581:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978587:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698143.978593:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146019072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698143.978605:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978607:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698143.978608:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978611:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978625:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698143.978627:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978629:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698143.978633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698143.978638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146019200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698143.978645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698143.978648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978650:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978652:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698143.978654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698143.978663:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146019328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698143.978669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace93100 x1874531146018816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698143.978674:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978678:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698143.978679:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978683:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978685:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698143.978686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146018816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698143.978690:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978694:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698143.978700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698143.978704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146019456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698143.978707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.978714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146018816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698143.978720:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698143.978724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146018944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698143.978725:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698143.978730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978736:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978741:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698143.978745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698143.978749:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698143.978758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace91880 x1874531146018944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698143.978761:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698143.978777:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146019584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698143.978780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146018944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698143.978792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698143.978797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698143.978803:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698143.978804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698143.978805:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978811:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978815:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698143.978815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146018944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698143.978821:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978825:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698143.978832:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146019712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698143.978843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146019072:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698143.978848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698143.978852:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698143.978853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698143.978857:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6043 00000100:00080000:0.0:1787698143.978862:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698143.978866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698143.978867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698143.978871:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698143.978881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace91500 x1874531146019072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698143.978882:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146019840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698143.978910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146019072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.978927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.978930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.978932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146019072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698143.978936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146019200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698143.978949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698143.978955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace92300 x1874531146019200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.978961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146019200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.978966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.978967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.978969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146019200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698143.978972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146019328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698143.978980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698143.978986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace93480 x1874531146019328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.978996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146019328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.979001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.979002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.979004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146019328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698143.979006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698143.979015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698143.979020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace92d80 x1874531146019456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.979026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146019456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.979032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.979034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.979037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698143.979042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698143.979058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698143.979068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace90e00 x1874531146019584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.979079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146019584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.979089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.979091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.979092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698143.979104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698143.979117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698143.979123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace91c00 x1874531146019712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.979129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146019712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.979134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.979136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.979138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698143.979140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698143.979148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698143.979153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff8f51ace90700 x1874531146019840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.979159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146019840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.979163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.979164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698143.979167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146019840:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698144.874512:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698145.898483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698146.922505:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698147.946505:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698148.970601:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698149.098460:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098471:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698149.098475:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098486:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098498:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098507:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698149.098551:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146019968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098579:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098581:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698149.098583:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098586:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098589:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098592:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098601:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698149.098609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146020096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098616:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098618:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698149.098619:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098621:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698149.098636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146019968:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698149.098638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146020224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698149.098656:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098660:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098668:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698149.098685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146020352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098691:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098694:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698149.098695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098697:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098700:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098703:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098705:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698149.098712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146020480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098718:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098720:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698149.098721:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098724:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098726:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098732:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098736:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698149.098744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146020608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098754:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698149.098759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698149.098759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698149.098761:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098768:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098771:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098773:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098776:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698149.098783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146020736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098792:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698149.098794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098796:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00080000:3.0:1787698149.098798:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098800:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098803:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698149.098811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698149.098814:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146020864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698149.098827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698149.098831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698149.098833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698149.098838:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6048 00000100:00100000:1.0:1787698149.098842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020352:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698149.098844:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698149.098852:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698149.098859:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698149.098872:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146020992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698149.098877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698149.098900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698149.098914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698149.098927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698149.098939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020992:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698149.994567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698151.018528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698152.042506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698152.170627:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787698152.170673:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698152.170688:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170730:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170744:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170753:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170762:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170777:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170785:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170792:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698152.170799:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698152.170854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f518738b800 x1874531146020864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.170880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146020864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.170891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.170897:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.170901:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.170910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f5187388380 x1874531146020992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.170921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146020992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.170928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.170932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.170935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.170941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f518738bb80 x1874531146020224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.170952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146020224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.170958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.170961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.170964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698152.170977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f5187389c00 x1874531146020352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.170988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146020352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.170996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.170999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.171003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.171009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f518738a300 x1874531146020480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.171018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146020480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.171024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.171027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.171030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698152.171037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f5187389500 x1874531146020608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.171047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146020608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.171056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.171059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.171062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698152.171068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f518738ad80 x1874531146020736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.171078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146020736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.171085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.171088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.171091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146020736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698152.171098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f518738aa00 x1874531146019968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.171109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146019968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.171150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.171154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.171157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146019968:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698152.171164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698149/real 1787698152] req@ffff8f5187388a80 x1874531146020096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698152.171176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146020096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698152.171183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698152.171186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698152.171189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146020096:192.168.201.125@tcp:38:kworker.0 00010000:00010000:1.0:1787698153.066476:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698154.090568:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698154.218463:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218473:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698154.218477:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698154.218486:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.218494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698154.218500:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698154.218526:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146021120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698154.218550:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218554:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698154.218556:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698154.218560:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.218565:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698154.218569:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698154.218578:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698154.218592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146021120:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698154.218593:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146021248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698154.218609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218612:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698154.218615:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698154.218619:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.218623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698154.218625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698154.218629:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698154.218642:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146021376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698154.218650:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698154.218655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698154.218658:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6053 00000800:00000100:1.0:1787698154.218663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698154.218664:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698154.218671:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698154.218677:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698154.218689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146021504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698154.218697:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218700:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698154.218702:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698154.218706:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.218710:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698154.218711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace90000 x1874531146021120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698154.218716:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698154.218725:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698154.218738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146021120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698154.218741:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146021632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698154.218768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218771:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698154.218774:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698154.218780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698154.218781:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.218788:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698154.218789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698154.218798:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698154.218799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146021120:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698154.218806:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698154.218821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146021760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698154.218830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698154.218831:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218847:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698154.218853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698154.218858:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6053 00000800:00000100:1.0:1787698154.218864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698154.218865:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698154.218873:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698154.218878:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698154.218882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace92680 x1874531146021248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698154.218894:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146021888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698154.218898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146021248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698154.218918:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698154.218920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698154.218926:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698154.218928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698154.218930:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698154.218935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698154.218939:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.218946:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698154.218947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021376:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698154.218953:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698154.218960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698154.218972:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146022016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698154.218977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698154.218988:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698154.218991:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698154.218994:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698154.218995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace91180 x1874531146021376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698154.219001:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6053 00000100:00080000:0.0:1787698154.219008:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698154.219014:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146021376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698154.219020:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698154.219033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698154.219036:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146022144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698154.219040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698154.219059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698154.219094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace90380 x1874531146021504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698154.219118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146021504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698154.219126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698154.219129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698154.219138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698154.219157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace93b80 x1874531146021632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698154.219179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146021632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698154.219186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698154.219189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146021632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698154.219209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146021760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698154.219229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace93800 x1874531146021760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698154.219251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146021760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698154.219258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698154.219261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146021760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698154.219268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146021888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698154.219284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace90000 x1874531146021888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698154.219305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146021888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698154.219312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698154.219315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146021888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698154.219324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146022016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698154.219339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace90700 x1874531146022016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698154.219388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146022016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698154.219400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698154.219402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146022016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698154.219410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146022144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698154.219442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698154.219454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698154/real 1787698154] req@ffff8f51ace91c00 x1874531146022144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698154.219464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146022144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698154.219471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698154.219473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698154.219476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146022144:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698155.114502:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698156.138530:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698157.162552:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698158.186526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698159.210494:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698159.338499:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338509:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698159.338513:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338523:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338533:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338540:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698159.338572:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146022272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698159.338597:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338601:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698159.338603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338610:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338614:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698159.338624:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698159.338637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146022400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698159.338642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146022272:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698159.338649:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338652:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698159.338655:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338662:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338665:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338668:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698159.338673:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698159.338683:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146022528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698159.338692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338694:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698159.338696:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338714:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338719:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698159.338724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698159.338726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698159.338737:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146022656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698159.338746:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338750:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698159.338752:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338755:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338759:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338763:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698159.338767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace90e00 x1874531146022272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698159.338769:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698159.338783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146022784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698159.338792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338795:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698159.338797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698159.338800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146022272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698159.338804:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338810:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338816:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698159.338821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698159.338822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698159.338829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698159.338836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146022912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698159.338844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146022272:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698159.338850:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338854:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698159.338856:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338860:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338863:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.338868:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698159.338870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022400:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698159.338874:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698159.338888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146023040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698159.338895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698159.338901:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698159.338906:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698159.338910:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698159.338910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace92d80 x1874531146022400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698159.338918:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.338925:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698159.338929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146022400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698159.338933:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698159.338940:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698159.338944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.338950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698159.338954:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146023168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698159.338958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022400:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698159.338972:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698159.338972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022528:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698159.338980:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698159.338983:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698159.338987:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6058 00000100:00080000:0.0:1787698159.339004:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698159.339009:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698159.339013:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698159.339018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698159.339029:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146023296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698159.339035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace93480 x1874531146022528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146022528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698159.339071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698159.339085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698159.339092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace92300 x1874531146022656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146022656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698159.339115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698159.339125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698159.339133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace91500 x1874531146022784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146022784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146022784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698159.339162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146022912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698159.339177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698159.339185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace91880 x1874531146022912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146022912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146022912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698159.339212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146023040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698159.339222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698159.339227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace93100 x1874531146023040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146023040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146023040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698159.339274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146023168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698159.339294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698159.339301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace90e00 x1874531146023168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146023168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146023168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698159.339324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146023296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698159.339348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698159.339391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698159/real 1787698159] req@ffff8f51ace91c00 x1874531146023296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698159.339403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146023296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698159.339411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698159.339414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698159.339417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146023296:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698160.234532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698161.258492:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698162.282494:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698163.306536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698164.330619:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698164.458490:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458501:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698164.458506:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458516:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458527:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458534:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698164.458578:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146023424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458610:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698164.458615:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458619:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458622:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458634:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698164.458644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146023552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458655:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698164.458658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458661:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458662:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458664:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698164.458673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146023680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458679:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698164.458683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458685:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458687:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458689:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458692:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698164.458697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146023424:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698164.458701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146023808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458712:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698164.458713:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458716:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458718:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458720:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458722:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698164.458729:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146023936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698164.458737:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458740:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458742:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458744:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458746:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698164.458752:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146024064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698164.458760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458762:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458772:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458775:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458777:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698164.458785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698164.458787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146024192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458797:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698164.458801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458804:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00080000:0.0:1787698164.458812:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458814:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698164.458824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146024320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698164.458830:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698164.458832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698164.458834:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698164.458837:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6064 00000100:00100000:1.0:1787698164.458837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace90700 x1874531146023424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698164.458839:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698164.458844:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698164.458850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698164.458858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146024448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698164.458864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146023424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.458874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.458877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.458880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146023424:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698164.458887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698164.458927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.458936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace90000 x1874531146023552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.458944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146023552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.458949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.458951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.458953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698164.458957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698164.458967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.458972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace93800 x1874531146023680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.458979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146023680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.458983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.458985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.458986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698164.459006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698164.459017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.459022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace93b80 x1874531146023808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.459039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146023808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.459044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.459046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.459048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698164.459050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698164.459060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.459065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace90380 x1874531146023936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.459072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146023936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.459076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.459078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.459079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146023936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698164.459082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698164.459091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.459096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace91180 x1874531146024064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.459102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146024064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.459111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.459113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.459115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698164.459117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698164.459126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.459131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace92680 x1874531146024192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.459137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146024192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.459142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.459143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.459145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698164.459147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698164.459156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.459161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace91c00 x1874531146024320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.459167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146024320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.459171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.459173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.459175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698164.459179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698164.459187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698164.459191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698164/real 1787698164] req@ffff8f51ace90e00 x1874531146024448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698164.459197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146024448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698164.459202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698164.459203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698164.459205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146024448:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698165.354546:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698166.378544:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698167.402480:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698168.426732:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698169.450582:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698169.578455:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578468:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698169.578478:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578491:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578506:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578515:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698169.578558:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146024576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698169.578580:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578583:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698169.578584:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578588:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578591:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578594:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578604:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698169.578614:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146024704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698169.578622:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698169.578624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146024576:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698169.578625:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698169.578627:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578633:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578638:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578641:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698169.578648:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146024832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698169.578658:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698169.578661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578663:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578667:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698169.578677:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146024960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698169.578682:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698169.578685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578687:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578704:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578707:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578710:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698169.578717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146025088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698169.578721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698169.578729:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578732:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698169.578733:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578737:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578741:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578746:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698169.578762:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146025216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698169.578771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698169.578777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578782:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00100000:1.0:1787698169.578782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace93100 x1874531146024576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698169.578789:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698169.578797:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698169.578812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146024576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698169.578827:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146025344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698169.578847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698169.578850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698169.578853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698169.578858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698169.578859:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578866:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698169.578867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146024576:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698169.578874:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578880:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698169.578892:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146025472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698169.578900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146024704:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698169.578916:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698169.578920:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698169.578922:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698169.578926:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6069 00000100:00080000:0.0:1787698169.578931:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698169.578933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698169.578937:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698169.578943:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698169.578951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace91880 x1874531146024704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698169.578966:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146025600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698169.578968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146024704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.578989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.578994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.578999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146024704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698169.579006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146024832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698169.579025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace91500 x1874531146024832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146024832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146024832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698169.579065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146024960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698169.579084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace92300 x1874531146024960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146024960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146024960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698169.579133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146025088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698169.579152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace93480 x1874531146025088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146025088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146025088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698169.579191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698169.579209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace92d80 x1874531146025216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146025216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698169.579256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698169.579274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace90e00 x1874531146025344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146025344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698169.579325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698169.579344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace91c00 x1874531146025472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146025472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698169.579419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698169.579437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698169.579448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698169/real 1787698169] req@ffff8f51ace92680 x1874531146025600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698169.579472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146025600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698169.579481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698169.579484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698169.579487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146025600:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698170.474598:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698171.498612:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698172.522541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698173.546534:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698174.570507:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698174.698482:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.698490:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698174.698494:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698503:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698512:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698519:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698174.698554:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146025728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.698584:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.698587:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698174.698589:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698593:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698596:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698599:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698608:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698174.698618:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146025856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.698628:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.698631:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698174.698633:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698638:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698640:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698645:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698649:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698174.698660:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146025984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.698669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.698672:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698174.698673:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698677:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698682:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698174.698686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146025728:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698174.698694:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698174.698715:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146026112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.698723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.698739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698174.698741:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698746:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698749:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698753:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698758:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698174.698770:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146026240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.698780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.698785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787698174.698793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698174.698793:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698809:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698816:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698821:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698174.698842:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146026368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.698852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698174.698858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f518738a680 x1874531146025728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698174.698859:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698174.698863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.698876:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698888:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698895:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698901:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698174.698909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146025728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698174.698915:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146026496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698174.698941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698174.698943:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698174.698948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698174.698950:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698174.698954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146025728:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698174.698955:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698174.698969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146025856:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698174.698970:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00080000:3.0:1787698174.698985:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.698993:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.698999:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698174.699013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698174.699015:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146026624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698174.699027:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698174.699030:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698174.699031:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698174.699039:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6074 00000100:00100000:1.0:1787698174.699040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f5187388000 x1874531146025856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698174.699042:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698174.699047:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698174.699052:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698174.699068:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146026752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698174.699076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146025856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146025856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698174.699120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146025984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698174.699138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f518738b480 x1874531146025984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146025984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146025984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698174.699165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146026112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698174.699175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f5187389f80 x1874531146026112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146026112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146026112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698174.699203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146026240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698174.699213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f5187388e00 x1874531146026240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146026240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146026240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698174.699259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698174.699289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f518738b100 x1874531146026368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146026368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698174.699336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698174.699404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f5187389180 x1874531146026496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146026496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698174.699452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698174.699468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f5187388a80 x1874531146026624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146026624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698174.699496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698174.699505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698174.699510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698174/real 1787698174] req@ffff8f518738aa00 x1874531146026752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698174.699516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146026752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698174.699521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698174.699523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698174.699524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146026752:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698175.594492:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698176.618566:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698177.642541:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698178.666549:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698179.690540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698179.818460:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818470:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698179.818474:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698179.818481:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818489:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818494:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698179.818522:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146026880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698179.818540:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818543:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698179.818544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698179.818547:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818549:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818551:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818558:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698179.818566:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146027008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698179.818572:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818575:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698179.818578:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698179.818579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146026880:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698179.818586:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818590:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818595:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818600:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698179.818612:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146027136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698179.818621:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818625:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698179.818626:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698179.818629:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818634:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818637:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818652:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698179.818660:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146027264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698179.818664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698179.818670:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818673:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698179.818674:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698179.818678:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818687:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818689:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698179.818699:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146027392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698179.818705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818707:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698179.818710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698179.818711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace91180 x1874531146026880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698179.818714:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818719:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818725:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698179.818735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146026880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698179.818740:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146027520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698179.818745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698179.818750:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698179.818751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698179.818753:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698179.818756:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698179.818759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146026880:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698179.818760:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818767:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698179.818772:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818777:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698179.818787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698179.818789:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146027648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698179.818806:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698179.818811:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698179.818814:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:0.0:1787698179.818818:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698179.818820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698179.818823:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818828:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698179.818834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace90380 x1874531146027008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698179.818839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146027776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698179.818848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146027008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698179.818849:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698179.818855:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698179.818858:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698179.818864:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6079 00000100:00080000:1.0:1787698179.818864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698179.818868:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698179.818870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698179.818874:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698179.818878:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698179.818878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698179.818888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146027904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698179.818891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698179.818926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.818939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace93b80 x1874531146027136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.818951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146027136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.818959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.818962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.818964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698179.818970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698179.818985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.818992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace93800 x1874531146027264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.819003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146027264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.819008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.819009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.819011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698179.819014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698179.819023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.819028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace90000 x1874531146027392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.819034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146027392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.819038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.819039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.819041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146027392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698179.819044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698179.819052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.819057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace90700 x1874531146027520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.819063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146027520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.819071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.819073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.819075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698179.819086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698179.819094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.819099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace91180 x1874531146027648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.819105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146027648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.819118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.819119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.819121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698179.819124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698179.819134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.819139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace92680 x1874531146027776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.819146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146027776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.819152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.819155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.819159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698179.819163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698179.819179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698179.819190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698179/real 1787698179] req@ffff8f51ace91c00 x1874531146027904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698179.819201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146027904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698179.819206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698179.819207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698179.819209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146027904:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698180.714539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698181.738525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698182.762527:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698183.786541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698184.810537:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698184.938468:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938476:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698184.938479:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938486:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938500:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698184.938534:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146028032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698184.938595:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938599:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938602:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938605:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938613:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698184.938627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146028160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698184.938636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146028032:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698184.938640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698184.938647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938652:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938656:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938659:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938662:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698184.938671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146028288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698184.938681:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938683:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938686:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938688:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938691:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698184.938698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698184.938699:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146028416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938711:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938713:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698184.938714:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938717:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938724:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938727:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698184.938734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146028544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938740:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698184.938743:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698184.938747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace90e00 x1874531146028032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698184.938749:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938754:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938759:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938764:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698184.938771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146028032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698184.938773:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146028672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938793:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698184.938793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698184.938799:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698184.938800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698184.938801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938806:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00100000:1.0:1787698184.938806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146028032:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698184.938811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938815:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698184.938823:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146028800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698184.938832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938835:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00100000:1.0:1787698184.938838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028160:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698184.938843:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698184.938848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938851:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698184.938858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146028928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698184.938864:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.938866:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698184.938867:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.938870:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6084 00000100:00080000:0.0:1787698184.938873:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698184.938878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698184.938879:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698184.938885:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698184.938897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace92d80 x1874531146028160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698184.938901:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146029056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698184.938933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146028160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.938941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.938945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.938949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698184.938956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698184.938977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.938989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace93480 x1874531146028288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.938999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146028288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698184.939018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698184.939036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.939045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace92300 x1874531146028416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.939070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146028416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698184.939082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698184.939095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.939101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace91500 x1874531146028544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.939107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146028544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146028544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698184.939118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146028672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698184.939127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.939132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace91880 x1874531146028672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.939138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146028672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146028672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698184.939161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146028800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698184.939169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.939175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace93100 x1874531146028800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.939181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146028800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146028800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698184.939191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146028928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698184.939199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.939204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace90e00 x1874531146028928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.939217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146028928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146028928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698184.939228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146029056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698184.939238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698184.939243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff8f51ace91c00 x1874531146029056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.939249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146029056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.939254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.939256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698184.939257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146029056:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698185.834560:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698186.858483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698187.882551:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698188.906514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698189.930508:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698190.058428:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058434:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698190.058437:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058445:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058453:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058458:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698190.058506:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146029184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698190.058528:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058530:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698190.058531:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058534:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058537:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058539:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698190.058545:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698190.058553:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146029312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698190.058559:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058561:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698190.058562:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058565:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058566:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058570:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698190.058572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146029184:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698190.058573:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698190.058583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146029440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698190.058588:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058590:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698190.058590:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058593:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058595:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058597:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698190.058600:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698190.058613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146029568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698190.058618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698190.058621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058623:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058626:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058628:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698190.058631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698190.058636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698190.058640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146029696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698190.058646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698190.058650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058653:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058658:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058660:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698190.058663:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698190.058670:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146029824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698190.058675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace92680 x1874531146029184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698190.058692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058695:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698190.058696:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698190.058700:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698190.058704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146029184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698190.058706:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698190.058711:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698190.058722:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146029952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698190.058723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.058733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698190.058734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698190.058741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146029184:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698190.058742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698190.058746:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698190.058753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029312:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698190.058754:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058762:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058767:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698190.058772:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698190.058778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698190.058787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146030080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698190.058791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace91180 x1874531146029312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.058814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146029312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698190.058815:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698190.058824:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698190.058829:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698190.058830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.058837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.058839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029312:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698190.058847:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6089 00000100:00080000:0.0:1787698190.058852:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698190.058858:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698190.058862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029440:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698190.058864:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698190.058882:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146030208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698190.058900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.058911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace90700 x1874531146029440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.058926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146029440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.058934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.058937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.058940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698190.058946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698190.058966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.058977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace90000 x1874531146029568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.058995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146029568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.059002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.059005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.059008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698190.059014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698190.059032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.059042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace93800 x1874531146029696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.059053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146029696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.059058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.059060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.059062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146029696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698190.059065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146029824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698190.059077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.059083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace93b80 x1874531146029824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.059090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146029824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.059099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.059101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.059102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146029824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698190.059105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146029952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698190.059113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.059118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace90380 x1874531146029952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.059125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146029952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.059132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.059135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.059138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146029952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698190.059143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146030080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698190.059160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.059170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace91c00 x1874531146030080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.059181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146030080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.059188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.059191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.059195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146030080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698190.059213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146030208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698190.059228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698190.059236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698190/real 1787698190] req@ffff8f51ace90e00 x1874531146030208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698190.059243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146030208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698190.059248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698190.059250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698190.059251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146030208:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698190.954557:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698191.978534:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698193.002600:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698194.026566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698195.050561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698195.178485:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178494:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698195.178498:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178507:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178519:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698195.178527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698195.178561:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146030336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698195.178580:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178583:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698195.178584:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178587:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178589:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698195.178592:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178599:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698195.178607:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146030464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698195.178614:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698195.178618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178622:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178625:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698195.178626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146030336:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698195.178627:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698195.178639:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146030592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698195.178645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698195.178649:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178652:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178655:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698195.178658:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178663:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698195.178674:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146030720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698195.178682:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178685:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698195.178686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178690:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6094 00000800:00000100:1.0:1787698195.178694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698195.178696:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698195.178702:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698195.178717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146030848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698195.178726:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698195.178730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178735:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178738:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698195.178742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace93100 x1874531146030336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698195.178744:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698195.178767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146030336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698195.178768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146030976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698195.178787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698195.178789:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698195.178795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698195.178796:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698195.178800:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698195.178803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146030336:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698195.178807:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178813:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698195.178817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698195.178829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698195.178834:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146031104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698195.178865:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698195.178870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698195.178871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698195.178875:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178880:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178885:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698195.178888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace91880 x1874531146030464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698195.178892:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178899:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698195.178907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146030464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698195.178913:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146031232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698195.178918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.178926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.178931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698195.178932:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698195.178939:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698195.178942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698195.178943:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698195.178953:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6094 00000100:00080000:0.0:1787698195.178958:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698195.178962:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698195.178967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698195.178968:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698195.178981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace91500 x1874531146030592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698195.178984:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146031360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698195.179005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146030592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698195.179027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698195.179043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698195.179052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace92300 x1874531146030720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698195.179069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146030720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698195.179087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698195.179108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698195.179119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace93480 x1874531146030848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698195.179131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146030848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146030848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698195.179150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146030976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698195.179167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698195.179178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace92d80 x1874531146030976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698195.179189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146030976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146030976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698195.179222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146031104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698195.179237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698195.179246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace93100 x1874531146031104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698195.179255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146031104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146031104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698195.179270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146031232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698195.179282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698195.179289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace90e00 x1874531146031232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698195.179298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146031232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146031232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698195.179314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146031360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698195.179325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698195.179333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698195/real 1787698195] req@ffff8f51ace91c00 x1874531146031360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698195.179341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146031360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698195.179374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698195.179392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698195.179400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146031360:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698196.074531:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698197.098523:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698198.122539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698199.146533:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698200.170499:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698200.298448:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298469:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698200.298474:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698200.298484:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298493:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698200.298500:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698200.298534:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146031488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698200.298564:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698200.298570:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698200.298575:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298578:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698200.298582:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698200.298591:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698200.298603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146031616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698200.298614:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298618:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698200.298622:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698200.298622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146031488:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698200.298632:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698200.298641:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698200.298646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698200.298658:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146031744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698200.298673:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298676:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698200.298678:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698200.298681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698200.298685:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298691:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698200.298695:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698200.298700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698200.298712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146031872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698200.298718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace90380 x1874531146031488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.298726:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698200.298732:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698200.298736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146031488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.298737:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298745:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698200.298748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698200.298752:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698200.298754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787698200.298762:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698200.298768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146031488:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698200.298776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146032000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698200.298787:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698200.298791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146031616:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698200.298792:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698200.298796:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698200.298802:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298806:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698200.298810:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698200.298815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698200.298819:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698200.298826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace93b80 x1874531146031616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698200.298833:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146032128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698200.298838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146031616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.298850:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698200.298851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698200.298857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698200.298859:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698200.298861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146031616:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698200.298863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698200.298868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146031744:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698200.298869:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.298874:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698200.298879:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698200.298884:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698200.298893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698200.298899:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146032256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698200.298907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace93800 x1874531146031744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.298910:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298920:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698200.298924:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698200.298931:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00100000:1.0:1787698200.298932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146031744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.298937:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698200.298941:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698200.298945:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698200.298946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.298952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146031744:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698200.298954:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698200.298958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146031872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698200.298970:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146032384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698200.298978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698200.298988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace90000 x1874531146031872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.298990:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698200.298998:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698200.299001:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698200.299008:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6099 00000100:00080000:0.0:1787698200.299015:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698200.299018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146031872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698200.299022:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698200.299029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698200.299030:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698200.299036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.299040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146031872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698200.299045:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146032512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698200.299049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146032000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698200.299078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698200.299089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace90700 x1874531146032000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698200.299096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146032000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698200.299101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698200.299103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.299105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146032000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698200.299115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698200.299126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698200.299131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace91180 x1874531146032128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698200.299137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146032128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698200.299145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698200.299147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.299149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698200.299152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698200.299159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698200.299164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace92680 x1874531146032256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698200.299170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146032256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698200.299174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698200.299176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.299177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698200.299180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698200.299187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698200.299192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace90380 x1874531146032384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698200.299198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146032384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698200.299202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698200.299205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.299208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698200.299213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698200.299228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698200.299238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698200/real 1787698200] req@ffff8f51ace91c00 x1874531146032512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698200.299248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146032512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698200.299256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698200.299259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698200.299261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146032512:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698201.194526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698202.218507:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698203.242555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698204.266504:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698205.290511:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698205.418449:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698205.418459:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698205.418463:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418472:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418482:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418489:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698205.418524:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146032640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698205.418556:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698205.418560:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698205.418561:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418566:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418569:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418574:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418582:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698205.418592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146032640:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698205.418596:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146032768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698205.418608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698205.418611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698205.418613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418616:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418620:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418631:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418635:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698205.418645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698205.418649:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146032896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698205.418659:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698205.418662:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698205.418664:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418676:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418679:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418686:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698205.418689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f518738ad80 x1874531146032640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698205.418692:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698205.418709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146032640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698205.418709:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146033024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698205.418738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698205.418739:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698205.418743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698205.418746:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698205.418749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146032640:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698205.418750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418758:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418761:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418764:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698205.418770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146032768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698205.418777:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146033152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698205.418791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698205.418794:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698205.418798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698205.418799:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418803:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00100000:1.0:1787698205.418804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f5187389500 x1874531146032768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698205.418806:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418811:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418818:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698205.418822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146032768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698205.418829:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146033280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698205.418830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.418837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698205.418841:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698205.418842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146032768:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698205.418845:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698205.418848:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698205.418850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146032896:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698205.418853:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418857:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418859:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418868:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698205.418872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698205.418880:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146033408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698205.418886:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698205.418889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f518738a300 x1874531146032896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698205.418890:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698205.418891:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698205.418896:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418900:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418905:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418909:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698205.418916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146032896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698205.418917:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146033536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698205.418940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698205.418943:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698205.418947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698205.418948:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698205.418951:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698205.418955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146032896:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698205.418956:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6105 00000100:00080000:3.0:1787698205.418961:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698205.418964:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698205.418967:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698205.418968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146033024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698205.418974:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146033664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698205.418993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698205.419006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f5187389c00 x1874531146033024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698205.419024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146033024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698205.419032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.419035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698205.419037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146033024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698205.419050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146033152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698205.419068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698205.419079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f518738bb80 x1874531146033152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698205.419089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146033152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698205.419096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.419099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698205.419102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146033152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698205.419108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698205.419125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698205.419136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f5187388380 x1874531146033280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698205.419147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146033280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698205.419160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.419163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698205.419166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698205.419171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698205.419188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698205.419198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f518738b800 x1874531146033408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698205.419222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146033408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698205.419229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.419232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698205.419235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698205.419242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698205.419260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698205.419272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f518738ad80 x1874531146033536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698205.419282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146033536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698205.419289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.419292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698205.419295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698205.419302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698205.419320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698205.419330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698205/real 1787698205] req@ffff8f518738aa00 x1874531146033664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698205.419341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146033664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698205.419349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698205.419351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698205.419382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146033664:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698206.314483:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698207.338534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698208.362509:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698209.386560:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698210.410523:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698210.538448:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538458:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698210.538462:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698210.538470:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538481:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538488:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698210.538522:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146033792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698210.538548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538552:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698210.538554:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698210.538558:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538562:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538566:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698210.538575:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698210.538585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146033792:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698210.538591:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146033920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698210.538603:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538607:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698210.538609:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698210.538612:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538616:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538621:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698210.538625:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698210.538637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698210.538640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146034048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698210.538651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538655:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698210.538657:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698210.538660:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538664:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538668:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698210.538674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698210.538687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f5187388a80 x1874531146033792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698210.538687:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146034176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698210.538719:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538723:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698210.538725:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698210.538730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146033792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698210.538733:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538739:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538746:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698210.538749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698210.538751:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698210.538757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.538762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146033792:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698210.538768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146034304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698210.538779:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538782:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698210.538784:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698210.538790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146033920:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698210.538791:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538798:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538804:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698210.538808:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698210.538822:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146034432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698210.538823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698210.538840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f5187389180 x1874531146033920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698210.538843:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538852:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698210.538855:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698210.538858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146033920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698210.538862:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.538870:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538875:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698210.538881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698210.538883:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698210.538888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.538893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146033920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698210.538899:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146034560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698210.538904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146034048:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698210.538917:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698210.538921:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698210.538923:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698210.538928:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6110 00000800:00000100:1.0:1787698210.538930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698210.538936:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.538943:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698210.538944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f518738b100 x1874531146034048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698210.538957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698210.538961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146034048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698210.538972:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146034688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698210.538977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.538986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698210.538987:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698210.538993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146034048:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698210.538994:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698210.538999:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698210.539004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146034176:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698210.539006:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6110 00000100:00080000:3.0:1787698210.539013:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698210.539019:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698210.539024:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698210.539035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698210.539038:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146034816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698210.539051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f5187388e00 x1874531146034176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698210.539081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146034176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698210.539089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.539092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.539095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146034176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698210.539101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146034304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698210.539132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698210.539143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f5187389f80 x1874531146034304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698210.539155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146034304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698210.539162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.539165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.539168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146034304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698210.539185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698210.539201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698210.539211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f518738b480 x1874531146034432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698210.539221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146034432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698210.539243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.539246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.539249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698210.539255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698210.539272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698210.539283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f5187388000 x1874531146034560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698210.539293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146034560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698210.539301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.539304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.539307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698210.539312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698210.539329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698210.539339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f518738a680 x1874531146034688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698210.539350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146034688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698210.539380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.539383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.539386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698210.539391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698210.539406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698210.539416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698210/real 1787698210] req@ffff8f5187388a80 x1874531146034816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698210.539425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146034816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698210.539431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698210.539434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698210.539437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146034816:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698211.434551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698212.458507:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698213.482488:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698214.506522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698215.530550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698215.658536:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.658551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698215.658556:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.658570:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.658593:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.658602:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698215.658660:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146034944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.658693:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.658699:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698215.658701:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.658705:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.658709:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.658714:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698215.658726:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698215.658742:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146035072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.658753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.658757:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698215.658759:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.658764:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.658771:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698215.658774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146034944:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698215.658781:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698215.658790:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698215.658808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146035200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.658820:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.658824:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698215.658826:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.658830:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.658835:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.658839:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698215.658846:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698215.658858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146035328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.658868:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.658872:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698215.658874:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.658879:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.658883:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.658889:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698215.658896:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698215.658902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698215.658916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146035456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.658928:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.658932:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698215.658934:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.658939:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.658944:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.658949:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698215.658957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698215.658957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698215.658988:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146035584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698215.658995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035328:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698215.659018:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698215.659019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035456:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698215.659043:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698215.659047:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.659054:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.659059:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.659065:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698215.659068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035584:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698215.659078:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698215.659100:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146035712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.659116:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.659120:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698215.659122:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.659127:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.659132:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698215.659137:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698215.659141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035712:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698215.659145:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698215.659167:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146035840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698215.659182:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698215.659186:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698215.659188:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698215.659193:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6115 00000100:00080000:0.0:1787698215.659199:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698215.659199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035840:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698215.659210:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698215.659217:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698215.659236:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146035968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698215.659278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035968:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698216.554539:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698217.578536:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698218.602440:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698218.730632:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787698218.730675:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698218.730691:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698218.730741:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787698218.730758:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698218.730768:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698218.730777:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698218.730783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace90380 x1874531146035968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698218.730791:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698218.730806:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698218.730820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146035968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698218.730821:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698218.730833:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787698218.730837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.730844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.730850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698218.730858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace91c00 x1874531146035840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.730871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146035840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.730880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.730883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.730886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698218.730893:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace91880 x1874531146035712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.730904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146035712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.730912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.730916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.730919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698218.730925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace91500 x1874531146035584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.730937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146035584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.730945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.730949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.730952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146035584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698218.730959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace92d80 x1874531146035200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.730970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146035200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.730994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.730997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.731001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698218.731007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace93480 x1874531146035328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.731019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146035328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.731027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.731032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.731035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698218.731042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace92300 x1874531146035456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.731053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146035456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.731061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.731065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.731069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698218.731075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace90e00 x1874531146034944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.731087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146034944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.731101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.731104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.731107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146034944:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698218.731114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698215/real 1787698218] req@ffff8f51ace93100 x1874531146035072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698218.731126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146035072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698218.731134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698218.731138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698218.731141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146035072:192.168.201.125@tcp:38:kworker.0 00010000:00010000:0.0:1787698219.626542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698220.650535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698220.778473:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698220.778485:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698220.778490:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778508:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778520:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698220.778560:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146036096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698220.778587:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698220.778591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698220.778593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778601:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778605:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778614:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698220.778628:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146036224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698220.778638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146036096:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698220.778641:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698220.778647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698220.778650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778654:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778658:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778662:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698220.778680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146036352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698220.778690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698220.778693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698220.778695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6120 00000800:00000100:1.0:1787698220.778705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698220.778707:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778713:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698220.778732:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146036480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698220.778741:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698220.778745:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698220.778749:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698220.778750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace92680 x1874531146036096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698220.778756:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778762:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778768:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778773:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698220.778778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146036096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698220.778789:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146036608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698220.778790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.778799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.778806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146036096:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698220.778808:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698220.778813:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698220.778816:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778821:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778824:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778828:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778833:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698220.778834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698220.778849:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146036736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698220.778860:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698220.778862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698220.778867:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698220.778870:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778875:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778879:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698220.778879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace91180 x1874531146036224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698220.778886:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778895:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698220.778899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146036224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.778908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698220.778910:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146036864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698220.778915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.778922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036224:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698220.778930:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698220.778932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698220.778937:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698220.778954:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698220.778960:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:0.0:1787698220.778965:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698220.778969:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.778975:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698220.778976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.778990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace90700 x1874531146036352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698220.778992:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146036992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698220.779018:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698220.779020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146036352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698220.779024:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698220.779028:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698220.779035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698220.779037:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6120 00000100:00080000:1.0:1787698220.779043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698220.779044:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698220.779052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698220.779053:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698220.779062:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698220.779067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698220.779078:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146037120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698220.779092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.779105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace90000 x1874531146036480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698220.779117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146036480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.779125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.779127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.779130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698220.779136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698220.779156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.779167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace93800 x1874531146036608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698220.779182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146036608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.779189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.779191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.779194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146036608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698220.779211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146036736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698220.779234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.779245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace93b80 x1874531146036736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698220.779255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146036736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.779263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.779265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.779269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146036736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698220.779274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146036864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698220.779291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.779301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace92680 x1874531146036864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698220.779311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146036864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.779323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.779325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.779336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146036864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698220.779341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146036992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698220.779390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.779401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace93100 x1874531146036992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698220.779414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146036992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.779421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.779424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.779427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146036992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698220.779432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146037120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698220.779450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698220.779459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698220/real 1787698220] req@ffff8f51ace90e00 x1874531146037120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698220.779476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146037120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698220.779482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698220.779484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698220.779488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146037120:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698221.674532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698222.698566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698223.722590:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698224.746581:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698225.770470:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698225.898473:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898483:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698225.898486:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698225.898496:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:2.0:1787698225.898506:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698225.898513:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698225.898549:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146037248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698225.898577:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898581:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698225.898583:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698225.898587:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:2.0:1787698225.898590:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698225.898594:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698225.898604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698225.898617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146037248:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698225.898619:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146037376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698225.898633:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898636:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698225.898638:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698225.898641:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:2.0:1787698225.898645:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698225.898649:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698225.898654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698225.898665:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146037504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698225.898672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698225.898680:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898684:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698225.898686:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698225.898691:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:2.0:1787698225.898695:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698225.898698:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698225.898704:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698225.898710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704fb80 x1874531146037248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698225.898717:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146037632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698225.898728:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698225.898729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146037248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698225.898736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698225.898739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.898743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698225.898745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:1.0:1787698225.898749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698225.898751:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698225.898753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146037248:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698225.898759:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698225.898765:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698225.898773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698225.898779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146037760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698225.898790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698225.898791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898796:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698225.898798:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698225.898801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704df80 x1874531146037376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698225.898807:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:2.0:1787698225.898814:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698225.898816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146037376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698225.898822:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698225.898826:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698225.898830:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698225.898836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.898839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698225.898846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698225.898847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146037888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698225.898874:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698225.898882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698225.898887:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6125 00000800:00000100:1.0:1787698225.898891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698225.898892:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698225.898899:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698225.898903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704f800 x1874531146037504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698225.898906:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698225.898920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146037504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698225.898923:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146038016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698225.898934:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.898941:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698225.898945:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.898948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698225.898952:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00080000:1.0:1787698225.898956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698225.898958:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698225.898963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037504:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698225.898966:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698225.898974:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698225.898976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698225.898987:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146038144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698225.898997:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698225.899001:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698225.899002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698225.899006:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698225.899012:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6125 00000100:00100000:1.0:1787698225.899015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704ce00 x1874531146037632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698225.899019:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698225.899027:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698225.899029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146037632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698225.899033:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698225.899036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698225.899039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.899042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698225.899049:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146038272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698225.899057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698225.899071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698225.899079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704e680 x1874531146037760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698225.899090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146037760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.899095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698225.899096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.899098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146037760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698225.899101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146037888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698225.899109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698225.899115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704c000 x1874531146037888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698225.899121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146037888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.899126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698225.899127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.899129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146037888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698225.899132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146038016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698225.899140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698225.899145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704c380 x1874531146038016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698225.899151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146038016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.899159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698225.899160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.899162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146038016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698225.899164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146038144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698225.899173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698225.899178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704c700 x1874531146038144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698225.899183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146038144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.899187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698225.899188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.899190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146038144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698225.899198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146038272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698225.899206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698225.899211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff8f518704fb80 x1874531146038272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698225.899217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146038272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.899221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698225.899223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698225.899226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146038272:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698226.794531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698227.818546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698228.842543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698229.866585:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698230.890536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698231.018483:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018490:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698231.018493:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018501:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:0.0:1787698231.018509:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018514:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698231.018543:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146038400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698231.018562:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018564:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698231.018565:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018568:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:0.0:1787698231.018570:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018572:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698231.018579:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698231.018587:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146038528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698231.018595:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018598:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698231.018600:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018606:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00100000:1.0:1787698231.018608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146038400:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698231.018610:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018618:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698231.018623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698231.018636:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146038656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698231.018644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698231.018649:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018653:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:0.0:1787698231.018665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018669:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698231.018674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698231.018674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698231.018689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146038784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698231.018696:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018698:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698231.018699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018702:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:0.0:1787698231.018705:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018710:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698231.018711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace92300 x1874531146038400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698231.018714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698231.018731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146038912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698231.018739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146038400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698231.018740:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698231.018751:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018760:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:1.0:1787698231.018762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698231.018780:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018785:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698231.018790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787698231.018792:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698231.018795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146038400:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698231.018803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698231.018807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146039040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698231.018814:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018817:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698231.018818:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018821:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:0.0:1787698231.018824:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018828:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698231.018830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698231.018833:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698231.018843:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146039168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698231.018850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace93480 x1874531146038528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698231.018853:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698231.018863:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698231.018874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146038528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698231.018876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018887:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00080000:1.0:1787698231.018888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698231.018894:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698231.018895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.018900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038528:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698231.018904:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698231.018911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698231.018923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698231.018926:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146039296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698231.018940:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698231.018944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698231.018946:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698231.018950:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698231.018955:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6130 00000100:00100000:1.0:1787698231.018960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace92d80 x1874531146038656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698231.018961:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698231.018969:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698231.018977:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698231.018985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146038656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698231.018991:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146039424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698231.019008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698231.019039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698231.019060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698231.019071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace91500 x1874531146038784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698231.019081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146038784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698231.019088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698231.019099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698231.019112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698231.019120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace91880 x1874531146038912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698231.019134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146038912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698231.019141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146038912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698231.019150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698231.019164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698231.019172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace91c00 x1874531146039040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698231.019181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146039040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698231.019188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698231.019198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698231.019213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698231.019221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace90380 x1874531146039168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698231.019230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146039168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698231.019249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698231.019270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698231.019285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698231.019294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace93480 x1874531146039296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698231.019303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146039296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698231.019310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698231.019320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698231.019332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698231.019340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698231/real 1787698231] req@ffff8f51ace92300 x1874531146039424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698231.019349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146039424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698231.019376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698231.019379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698231.019382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146039424:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698231.914568:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698232.938499:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698233.962568:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698234.986537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698236.010532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698236.138576:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.138585:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698236.138588:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.138596:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.138605:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698236.138611:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698236.138644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146039552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698236.138670:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.138674:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698236.138690:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.138695:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.138700:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698236.138704:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698236.138712:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698236.138714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146039552:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698236.138724:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146039680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698236.138739:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.138742:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698236.138744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.138747:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.138765:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698236.138770:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698236.138774:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698236.138785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698236.138787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146039808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698236.138796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.138803:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698236.138805:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.138808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.138811:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698236.138814:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698236.138818:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698236.138826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183647800 x1874531146039552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698236.138830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146039936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698236.138842:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.138856:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698236.138862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146039552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698236.138866:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698236.138876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698236.138879:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:0.0:1787698236.138884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698236.138885:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698236.138890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146039552:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698236.138893:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698236.138900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698236.138913:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146040064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698236.138918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146039680:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698236.138928:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.138932:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698236.138934:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.138938:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.138942:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698236.138946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698236.138956:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698236.138962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183646680 x1874531146039680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698236.138966:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698236.138980:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146040192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698236.138982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146039680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698236.139001:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.139008:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698236.139010:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.139014:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.139017:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698236.139025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698236.139025:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698236.139033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787698236.139033:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698236.139040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146039680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698236.139049:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146040320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698236.139050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146039808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698236.139061:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698236.139073:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698236.139075:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698236.139079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698236.139083:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.139088:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698236.139093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183644000 x1874531146039808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698236.139094:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698236.139103:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698236.139111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146039808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698236.139115:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146040448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698236.139127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146039808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698236.139142:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698236.139144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146039936:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787698236.139151:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698236.139153:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698236.139158:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6135 00000100:00080000:1.0:1787698236.139164:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698236.139168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698236.139171:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698236.139178:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698236.139183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183644700 x1874531146039936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698236.139192:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146040576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698236.139197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146039936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698236.139213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146039936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698236.139237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146040064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698236.139268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698236.139279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183647480 x1874531146040064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698236.139289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146040064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698236.139296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146040064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698236.139307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698236.139322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698236.139332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183645c00 x1874531146040192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698236.139343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146040192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698236.139350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698236.139394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698236.139411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698236.139421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183645180 x1874531146040320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698236.139432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146040320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698236.139439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698236.139450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698236.139467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698236.139478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183647100 x1874531146040448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698236.139487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146040448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698236.139493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698236.139503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698236.139520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698236.139530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698236/real 1787698236] req@ffff8f5183644380 x1874531146040576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698236.139540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146040576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698236.139548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698236.139551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698236.139554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146040576:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698237.034599:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698238.058546:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698239.082532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698240.106528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698241.130508:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698241.258456:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258465:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698241.258469:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258478:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258488:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258494:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698241.258536:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146040704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258569:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258572:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698241.258574:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258587:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258614:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258618:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698241.258640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146040832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258658:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698241.258662:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258666:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258669:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258672:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258676:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698241.258686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146040960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698241.258704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258710:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258712:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698241.258724:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146041088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258739:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698241.258742:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258747:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258749:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698241.258759:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146041216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258764:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258766:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698241.258767:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258769:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258771:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258772:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258774:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698241.258779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146041344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258784:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698241.258787:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258788:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258790:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258793:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258795:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698241.258799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146041472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698241.258806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258807:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258809:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258810:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258812:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698241.258816:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146041600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698241.258821:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698241.258822:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698241.258823:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698241.258824:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6140 00000100:00080000:1.0:1787698241.258826:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698241.258828:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698241.258830:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698241.258835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146041728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698241.258928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146040704:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787698241.258986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183644e00 x1874531146040704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146040704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146040704:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698241.259159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146040832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698241.259182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183646d80 x1874531146040832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146040832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146040832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698241.259239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146040960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698241.259261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183646a00 x1874531146040960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146040960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146040960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698241.259287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146041088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698241.259296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183647b80 x1874531146041088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146041088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146041088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698241.259338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146041216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698241.259384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183644380 x1874531146041216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146041216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146041216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698241.259412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698241.259423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183647100 x1874531146041344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146041344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698241.259445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698241.259453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183645180 x1874531146041472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146041472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698241.259475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698241.259482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183645c00 x1874531146041600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146041600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698241.259507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698241.259523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698241.259533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698241/real 1787698241] req@ffff8f5183647480 x1874531146041728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698241.259544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146041728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698241.259552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698241.259554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698241.259557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146041728:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698242.154537:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698243.178474:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698244.202570:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698245.226558:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698246.250482:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698246.378459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378466:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698246.378468:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378484:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378493:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698246.378498:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698246.378550:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146041856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698246.378579:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378583:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698246.378585:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378589:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378593:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698246.378603:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698246.378613:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698246.378628:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146041984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698246.378630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146041856:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698246.378646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698246.378651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378655:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698246.378662:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698246.378667:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698246.378678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146042112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698246.378687:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378690:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698246.378692:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698246.378697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698246.378700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378705:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698246.378710:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698246.378715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698246.378727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146042240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698246.378734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f518738aa00 x1874531146041856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698246.378738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698246.378751:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698246.378755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146041856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698246.378757:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378763:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698246.378769:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698246.378773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698246.378775:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698246.378781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.378789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146041856:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698246.378789:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146042368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698246.378804:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378808:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698246.378810:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378815:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378818:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698246.378822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146041984:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698246.378824:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698246.378830:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698246.378841:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146042496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698246.378852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378855:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698246.378857:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378860:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6146 00000800:00000100:0.0:1787698246.378867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698246.378877:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698246.378883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f518738ad80 x1874531146041984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698246.378883:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698246.378889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698246.378904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146042624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698246.378908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146041984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698246.378925:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378928:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698246.378930:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378938:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:0.0:1787698246.378939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698246.378941:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698246.378947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698246.378949:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698246.378954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146041984:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698246.378956:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698246.378964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146042112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698246.378970:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146042752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698246.378978:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698246.378980:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698246.378982:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698246.378985:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6146 00000100:00080000:3.0:1787698246.378989:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698246.378998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698246.379003:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698246.379011:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698246.379014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f518738b800 x1874531146042112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698246.379023:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146042880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698246.379031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146042112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146042112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698246.379065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146042240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698246.379086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698246.379097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f5187388380 x1874531146042240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698246.379109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146042240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146042240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698246.379128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146042368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698246.379145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698246.379156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f518738bb80 x1874531146042368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698246.379178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146042368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146042368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698246.379217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698246.379235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698246.379245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f5187389c00 x1874531146042496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698246.379254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146042496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698246.379273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698246.379291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698246.379302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f518738a300 x1874531146042624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698246.379319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146042624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698246.379336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698246.379391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698246.379403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f5187389500 x1874531146042752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698246.379415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146042752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698246.379434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698246.379452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698246.379471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698246/real 1787698246] req@ffff8f518738aa00 x1874531146042880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698246.379482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146042880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698246.379495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698246.379498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698246.379515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146042880:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698247.274563:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698248.298574:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698249.322523:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698250.346521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698251.372459:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698251.498494:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498505:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698251.498508:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698251.498519:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498529:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.498536:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698251.498573:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146043008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698251.498601:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498605:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698251.498607:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698251.498611:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498615:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.498619:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698251.498628:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698251.498641:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146043136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698251.498643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146043008:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698251.498659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498662:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698251.498664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698251.498668:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498672:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.498677:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698251.498682:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698251.498693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146043264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698251.498699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698251.498709:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698251.498714:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698251.498718:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498723:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.498727:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698251.498732:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698251.498734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704d180 x1874531146043008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698251.498745:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146043392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698251.498750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146043008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698251.498766:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498771:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698251.498772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698251.498774:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698251.498779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698251.498784:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00100000:0.0:1787698251.498788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146043008:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698251.498791:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.498799:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698251.498804:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698251.498815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698251.498818:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146043520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698251.498829:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498832:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698251.498834:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698251.498839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698251.498843:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498848:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698251.498856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704dc00 x1874531146043136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698251.498860:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698251.498871:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698251.498877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146043136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.498886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698251.498887:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146043648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698251.498894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.498902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043136:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698251.498909:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698251.498913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043264:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698251.498915:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698251.498920:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698251.498926:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498930:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.498934:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698251.498940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698251.498940:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698251.498953:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146043776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698251.498957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704ea00 x1874531146043264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698251.498972:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.498978:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698251.498981:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698251.498983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146043264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698251.498988:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6151 00000100:00080000:2.0:1787698251.498995:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698251.499000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698251.499001:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698251.499008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698251.499010:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698251.499012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698251.499019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698251.499022:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146043904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698251.499029:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698251.499032:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698251.499033:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698251.499036:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6151 00000800:00000100:0.0:1787698251.499040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698251.499042:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698251.499046:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698251.499050:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698251.499053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704e300 x1874531146043392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698251.499060:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146044032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698251.499066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146043392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.499076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698251.499079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.499082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698251.499099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698251.499119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698251.499129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704d880 x1874531146043520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698251.499137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146043520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.499148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698251.499151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.499153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146043520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698251.499156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146043648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698251.499169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698251.499175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704d500 x1874531146043648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698251.499181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146043648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.499186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698251.499188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.499189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146043648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698251.499192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146043776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698251.499201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698251.499206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704f480 x1874531146043776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698251.499217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146043776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.499221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698251.499222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.499224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146043776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698251.499227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146043904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698251.499235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698251.499240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704ed80 x1874531146043904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698251.499246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146043904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.499250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698251.499251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.499253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146043904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698251.499256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146044032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698251.499263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698251.499268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698251/real 1787698251] req@ffff8f518704d180 x1874531146044032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698251.499274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146044032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698251.499282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698251.499284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698251.499286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146044032:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698252.394504:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698253.418582:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698254.442743:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698255.466518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698256.490543:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698256.618425:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618435:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698256.618439:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618449:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618468:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618475:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698256.618513:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146044160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698256.618537:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618541:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698256.618542:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618547:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618550:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618554:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698256.618562:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698256.618576:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146044288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698256.618583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146044160:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698256.618598:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618601:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698256.618603:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618607:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618611:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618614:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698256.618619:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698256.618629:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146044416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698256.618638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698256.618645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698256.618645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618652:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618656:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618660:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698256.618665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698256.618675:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146044544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698256.618689:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698256.618690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704fb80 x1874531146044160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698256.618695:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698256.618698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00100000:0.0:1787698256.618718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146044160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698256.618720:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618728:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698256.618735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698256.618738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.618743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698256.618749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146044672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698256.618750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146044160:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698256.618763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698256.618768:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618772:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618775:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698256.618777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044288:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698256.618781:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698256.618786:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698256.618797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146044800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698256.618807:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618810:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787698256.618813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698256.618814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698256.618820:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618825:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618829:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698256.618832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704c700 x1874531146044288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698256.618841:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698256.618851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146044288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698256.618855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146044928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698256.618875:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618881:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698256.618884:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698256.618889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698256.618890:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618896:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698256.618898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698256.618902:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698256.618907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044288:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698256.618909:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698256.618919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698256.618923:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146045056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698256.618933:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698256.618937:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698256.618939:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698256.618944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698256.618945:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6156 00000100:00080000:2.0:1787698256.618951:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698256.618955:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698256.618960:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698256.618961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704c380 x1874531146044416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698256.618972:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146045184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698256.618978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146044416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698256.619028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698256.619049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698256.619058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704c000 x1874531146044544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698256.619070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146044544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698256.619089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698256.619102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698256.619112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704e680 x1874531146044672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698256.619122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146044672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146044672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698256.619150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146044800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698256.619165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698256.619183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704ce00 x1874531146044800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698256.619195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146044800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146044800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698256.619213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146044928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698256.619227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698256.619236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704f800 x1874531146044928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698256.619245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146044928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146044928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698256.619262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146045056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698256.619275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698256.619284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704df80 x1874531146045056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698256.619294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146045056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146045056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698256.619311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146045184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698256.619324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698256.619396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698256/real 1787698256] req@ffff8f518704fb80 x1874531146045184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698256.619407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146045184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698256.619415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698256.619418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698256.619421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146045184:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698257.514539:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698258.538542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698259.562495:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698260.586509:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698261.610937:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698261.738522:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.738532:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698261.738537:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738546:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738557:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698261.738564:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698261.738601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146045312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698261.738629:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.738633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698261.738635:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738640:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738644:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698261.738648:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698261.738658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698261.738672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146045312:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698261.738673:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146045440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698261.738694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.738698:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698261.738700:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738703:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738707:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698261.738709:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698261.738714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698261.738725:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146045568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698261.738732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698261.738744:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.738748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698261.738750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738754:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738758:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698261.738768:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698261.738775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f5187388a80 x1874531146045312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698261.738781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698261.738788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146045312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.738796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.738800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698261.738800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146045696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698261.738807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146045312:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698261.738815:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.738818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698261.738819:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738823:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738827:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698261.738831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045440:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698261.738835:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698261.738842:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698261.738855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146045824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698261.738857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698261.738870:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698261.738874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f518738a680 x1874531146045440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698261.738875:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698261.738878:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738884:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738889:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698261.738898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146045440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698261.738906:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698261.738914:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698261.738920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.738926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698261.738931:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146045952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698261.738932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698261.738946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698261.738949:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.738953:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698261.738955:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.738961:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.738964:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698261.738968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698261.738971:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698261.738977:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698261.738983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f5187388000 x1874531146045568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698261.738996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146045568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.739004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698261.739006:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146046080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698261.739010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698261.739017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698261.739025:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.739027:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698261.739029:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698261.739034:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.739039:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698261.739040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045696:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698261.739047:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698261.739054:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698261.739066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698261.739069:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146046208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698261.739082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f518738b480 x1874531146045696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698261.739087:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698261.739096:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698261.739100:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698261.739105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146045696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698261.739106:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6161 00000100:00080000:3.0:1787698261.739113:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698261.739119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.739123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698261.739128:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698261.739130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045696:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787698261.739137:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698261.739151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698261.739153:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146046336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698261.739191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698261.739203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f5187389f80 x1874531146045824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698261.739215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146045824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.739222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.739225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698261.739228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146045824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698261.739232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146045952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698261.739280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698261.739292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f5187388e00 x1874531146045952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698261.739302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146045952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.739309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.739312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698261.739315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146045952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698261.739320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146046080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698261.739336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698261.739406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f518738b100 x1874531146046080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698261.739418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146046080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.739425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.739427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698261.739429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146046080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698261.739433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146046208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698261.739447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698261.739454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f5187389180 x1874531146046208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698261.739460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146046208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.739464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.739466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698261.739467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146046208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698261.739477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146046336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698261.739487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698261.739492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698261/real 1787698261] req@ffff8f518738a680 x1874531146046336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698261.739498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146046336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698261.739503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698261.739505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698261.739506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146046336:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698262.634501:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698263.658504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698264.682472:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698265.706541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698266.730527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698266.858481:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858493:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698266.858498:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858508:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858520:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698266.858568:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146046464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698266.858594:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858597:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698266.858599:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858604:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858608:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698266.858631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698266.858644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146046592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698266.858647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146046464:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698266.858654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858656:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698266.858658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858661:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858665:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858668:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698266.858672:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698266.858682:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146046720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698266.858690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698266.858694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858697:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858702:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698266.858706:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698266.858715:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146046848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698266.858717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698266.858727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698266.858731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858735:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858738:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858741:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698266.858744:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698266.858753:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146046976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698266.858760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858763:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698266.858765:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698266.858768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f5187388a80 x1874531146046464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698266.858768:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858771:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858782:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698266.858785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146046464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698266.858787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698266.858796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698266.858796:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146047104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698266.858799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.858804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146046464:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698266.858804:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858808:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698266.858809:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858819:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858822:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858826:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698266.858828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046592:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698266.858830:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698266.858839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146047232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698266.858848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698266.858848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698266.858851:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698266.858853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858856:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00100000:0.0:1787698266.858859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f518738aa00 x1874531146046592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698266.858859:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858863:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698266.858867:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698266.858870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146046592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698266.858877:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146047360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698266.858878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.858882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698266.858885:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698266.858886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698266.858889:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698266.858891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698266.858891:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698266.858894:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6166 00000100:00080000:3.0:1787698266.858897:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698266.858900:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698266.858904:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698266.858906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698266.858912:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146047488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698266.858916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f5187389500 x1874531146046720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.858927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146046720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.858934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.858936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.858939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698266.858944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698266.858962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698266.858973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f518738a300 x1874531146046848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.858983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146046848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.858989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.858991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.858992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698266.858996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698266.859009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698266.859015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f5187389c00 x1874531146046976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.859021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146046976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.859026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.859027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.859029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146046976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698266.859034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698266.859044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698266.859050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f518738bb80 x1874531146047104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.859057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146047104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.859062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.859064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.859065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698266.859076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698266.859086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698266.859091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f5187388380 x1874531146047232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.859102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146047232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.859107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.859108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.859110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698266.859113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698266.859122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698266.859126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f518738b800 x1874531146047360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.859133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146047360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.859137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.859138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.859140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698266.859144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698266.859172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698266.859182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff8f518738ad80 x1874531146047488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.859193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146047488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.859206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698266.859208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698266.859211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146047488:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698267.754506:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698268.778540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698269.802515:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698270.826526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698271.850556:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698271.978500:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.978510:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698271.978515:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.978525:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978537:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978543:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698271.978585:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146047616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698271.978613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.978618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698271.978620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.978624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978629:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978633:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.978644:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698271.978659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146047744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698271.978669:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.978673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698271.978674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.978679:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978683:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978688:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.978695:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698271.978700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146047616:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698271.978709:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146047872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698271.978722:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.978725:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698271.978733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.978738:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978742:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978746:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.978754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698271.978766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146048000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698271.978777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698271.978778:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.978785:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698271.978787:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.978792:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978797:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978802:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.978808:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698271.978824:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146048128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698271.978831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704d180 x1874531146047616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698271.978837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.978852:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698271.978859:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.978867:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978873:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978879:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.978884:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698271.978888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146047616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.978904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698271.978905:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146048256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698271.978919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698271.978928:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698271.978930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146047616:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698271.978940:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698271.978944:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698271.978948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146047744:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698271.978954:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.978964:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.978971:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.978978:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698271.978990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698271.978997:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146048384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698271.979009:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.979013:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698271.979018:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698271.979019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704ed80 x1874531146047744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698271.979031:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.979042:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698271.979048:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146047744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698271.979079:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698271.979082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146047744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698271.979100:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146048512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698271.979114:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698271.979118:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698271.979120:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698271.979125:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6171 00000100:00080000:2.0:1787698271.979129:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698271.979132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146047872:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698271.979139:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698271.979147:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698271.979163:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146048640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698271.979171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704f480 x1874531146047872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146047872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146047872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698271.979221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146048000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698271.979235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704d500 x1874531146048000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146048000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146048000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698271.979271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146048128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698271.979289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704d880 x1874531146048128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146048128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146048128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698271.979317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698271.979329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704e300 x1874531146048256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146048256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698271.979434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698271.979461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704ea00 x1874531146048384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146048384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698271.979504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698271.979517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704dc00 x1874531146048512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146048512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698271.979570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698271.979587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698271.979594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff8f518704ed80 x1874531146048640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.979601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146048640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.979609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.979611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698271.979612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146048640:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698272.874536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698273.898493:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698274.922514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698275.946520:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698276.970517:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698277.098483:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098494:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698277.098498:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098508:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098519:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098526:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698277.098560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146048768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098581:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098583:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698277.098585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098588:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098590:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098593:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098600:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698277.098610:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146048896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098617:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698277.098622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146048768:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698277.098622:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698277.098624:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098631:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098634:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098643:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698277.098659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146049024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698277.098673:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098677:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098681:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098688:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698277.098689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698277.098703:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146049152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098710:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698277.098714:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098716:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098719:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098722:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698277.098732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704d180 x1874531146048768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698277.098733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146049280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098761:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698277.098763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698277.098764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146048768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698277.098770:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098777:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698277.098779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698277.098781:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098787:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698277.098788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.098794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146048768:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698277.098797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146049408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098805:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098807:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698277.098809:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098812:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098813:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098815:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098818:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698277.098823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146048896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698277.098826:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146049536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098834:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098836:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698277.098838:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098840:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098842:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698277.098844:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098847:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698277.098857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698277.098857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146049664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698277.098868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698277.098871:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698277.098874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704fb80 x1874531146048896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698277.098875:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698277.098883:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6176 00000100:00080000:2.0:1787698277.098890:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698277.098893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146048896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698277.098896:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698277.098905:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698277.098908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.098912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.098915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146048896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698277.098920:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146049792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698277.098923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146049024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698277.098944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.098952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704df80 x1874531146049024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.098961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146049024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.098969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.098972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.098975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146049024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698277.098992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146049152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698277.099012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.099022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704f800 x1874531146049152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.099029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146049152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.099034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.099036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.099039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146049152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698277.099042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146049280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698277.099054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.099059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704ce00 x1874531146049280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.099066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146049280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.099070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.099072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.099073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146049280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698277.099076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698277.099088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.099096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704e680 x1874531146049408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.099106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146049408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.099114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.099117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.099120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698277.099138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698277.099152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.099160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704c000 x1874531146049536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.099166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146049536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.099171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.099173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.099175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698277.099178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698277.099186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.099191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704c380 x1874531146049664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.099197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146049664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.099201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.099203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.099204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698277.099207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698277.099215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698277.099223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698277/real 1787698277] req@ffff8f518704c700 x1874531146049792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698277.099233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146049792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698277.099251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698277.099255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698277.099258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146049792:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698277.994497:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698279.018529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698280.042503:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698281.066487:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698282.090466:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698282.218452:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698282.218462:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218484:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218490:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698282.218518:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146049920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698282.218538:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218540:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698282.218542:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218545:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218552:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218555:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698282.218562:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698282.218572:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146050048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698282.218576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146049920:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698282.218582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218585:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698282.218586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218595:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218596:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698282.218599:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698282.218609:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146050176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698282.218624:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218626:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698282.218628:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218632:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218636:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218640:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698282.218644:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698282.218656:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146050304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698282.218662:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698282.218666:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218668:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218671:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218674:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698282.218677:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698282.218685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698282.218692:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146050432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698282.218701:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218704:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698282.218706:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218711:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00100000:0.0:1787698282.218714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050176:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698282.218717:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218724:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698282.218730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050304:192.168.201.125@tcp:8:kworker.0 00000008:00100000:2.0:1787698282.218730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698282.218746:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146050560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698282.218755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050432:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698282.218756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218763:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698282.218764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218768:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218771:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218774:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698282.218777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050560:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698282.218779:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698282.218793:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146050688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698282.218801:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218803:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698282.218806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698282.218807:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218813:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218822:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218825:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698282.218833:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698282.218844:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146050816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698282.218862:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698282.218865:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698282.218867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698282.218870:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6181 00000100:00080000:2.0:1787698282.218874:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698282.218878:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698282.218882:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698282.218887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698282.218920:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146050944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698282.218959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050944:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698283.114544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698284.138505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698285.162507:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698285.290628:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787698285.290671:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698285.290687:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698285.290736:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787698285.290750:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698285.290759:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698285.290768:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698285.290777:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698285.290783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704d180 x1874531146050944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698285.290786:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698285.290799:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698285.290815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146050944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698285.290816:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698285.290832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.290848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.290853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698285.290860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704fb80 x1874531146050816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.290872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146050816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.290879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.290883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.290886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698285.290893:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704df80 x1874531146050688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.290905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146050688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.290912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.290915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.290918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698285.290924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704f800 x1874531146050560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.290935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146050560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.290943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.290947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.290950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146050560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698285.290957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704ce00 x1874531146050432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.290967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146050432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.290974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.290976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.290979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698285.290987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704c380 x1874531146050048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.290996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146050048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.291003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.291007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.291009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698285.291015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704c000 x1874531146050176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.291025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146050176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.291031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.291034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.291037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698285.291043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704e680 x1874531146050304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.291052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146050304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.291058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.291061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.291063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146050304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698285.291068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698282/real 1787698285] req@ffff8f518704c700 x1874531146049920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698285.291078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146049920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698285.291084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698285.291087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698285.291090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146049920:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787698286.186555:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698287.210492:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698287.338465:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698287.338474:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698287.338478:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.338486:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.338496:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698287.338502:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698287.338534:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146051072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698287.338559:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698287.338565:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698287.338567:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.338571:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.338575:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698287.338578:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698287.338587:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698287.338600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146051072:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698287.338602:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146051200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698287.338616:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698287.338620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698287.338621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.338625:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.338628:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698287.338633:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698287.338637:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698287.338650:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146051328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698287.338658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698287.338665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698287.338669:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698287.338672:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.338676:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.338680:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698287.338684:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698287.338690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698287.338698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f518738ad80 x1874531146051072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698287.338705:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146051456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698287.338718:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698287.338722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146051072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698287.338723:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698287.338727:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.338737:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:0.0:1787698287.338741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698287.338742:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698287.338751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698287.338752:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698287.338759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146051072:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698287.338761:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698287.338778:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146051584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698287.338787:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698287.338797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051200:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698287.338798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698287.338814:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.338821:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.338826:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698287.338831:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698287.338837:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698287.338841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698287.338854:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146051712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698287.338859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f518738b800 x1874531146051200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698287.338877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698287.338883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146051200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698287.338884:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698287.338889:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698287.338898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698287.338898:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:0.0:1787698287.338905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698287.338906:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698287.338911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051200:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698287.338918:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698287.338922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051328:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698287.338925:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698287.338941:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146051840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698287.338949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698287.338961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f5187388380 x1874531146051328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698287.338968:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698287.338978:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698287.338981:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698287.338985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146051328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698287.338988:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.338994:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698287.339001:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698287.339010:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698287.339011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698287.339027:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146051968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698287.339032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051456:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698287.339050:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698287.339054:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698287.339058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698287.339058:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698287.339065:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6186 00000100:00080000:3.0:1787698287.339069:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698287.339074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f518738bb80 x1874531146051456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698287.339077:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698287.339087:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698287.339093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146051456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698287.339103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698287.339113:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146052096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698287.339125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698287.339148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698287.339159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f5187389c00 x1874531146051584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698287.339169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146051584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698287.339175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146051584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698287.339198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146051712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698287.339216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698287.339226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f518738a300 x1874531146051712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698287.339235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146051712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698287.339251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146051712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698287.339262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146051840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698287.339281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698287.339291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f5187389500 x1874531146051840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698287.339303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146051840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698287.339316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146051840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698287.339328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146051968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698287.339346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698287.339434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f518738aa00 x1874531146051968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698287.339446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146051968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698287.339454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146051968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698287.339468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146052096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698287.339489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698287.339499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698287/real 1787698287] req@ffff8f5187388a80 x1874531146052096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698287.339510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146052096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698287.339516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698287.339519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698287.339522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146052096:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698288.234533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698289.258513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698290.282532:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698291.306483:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698292.330489:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698292.458441:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458449:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698292.458453:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458461:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458471:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698292.458477:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698292.458518:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146052224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698292.458543:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458547:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698292.458548:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458552:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458555:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698292.458559:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458567:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698292.458578:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146052352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698292.458587:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698292.458591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146052224:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698292.458593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698292.458596:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458601:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458604:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698292.458608:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698292.458624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146052480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698292.458631:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698292.458635:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458638:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458642:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698292.458646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698292.458649:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458654:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698292.458665:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146052608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698292.458672:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698292.458677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458680:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458683:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698292.458686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f518738a680 x1874531146052224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698292.458689:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698292.458708:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146052736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698292.458740:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698292.458751:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458755:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458759:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698292.458763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146052224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698292.458764:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458772:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698292.458784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698292.458785:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146052864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698292.458792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.458797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146052224:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698292.458801:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698292.458806:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458810:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458813:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698292.458817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698292.458820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052352:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698292.458823:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698292.458836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146052992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698292.458845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698292.458845:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698292.458852:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458857:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6192 00000100:00080000:3.0:1787698292.458861:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698292.458864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f5187389180 x1874531146052352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698292.458866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458874:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698292.458879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146052352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.458887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698292.458888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146053120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698292.458892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.458899:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698292.458907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698292.458907:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698292.458914:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698292.458916:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698292.458921:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6192 00000800:00000100:0.0:1787698292.458925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698292.458925:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698292.458932:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698292.458938:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698292.458940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f518738b100 x1874531146052480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698292.458951:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146053248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698292.458956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146052480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.458974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.458977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.458982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698292.458988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698292.459007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698292.459017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f5187388e00 x1874531146052608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698292.459027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146052608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.459034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.459036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.459039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698292.459045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698292.459061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698292.459070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f5187389f80 x1874531146052736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698292.459081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146052736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.459087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.459090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.459093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146052736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698292.459098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146052864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698292.459115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698292.459126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f518738b480 x1874531146052864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698292.459137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146052864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.459144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.459147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.459150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146052864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698292.459165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146052992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698292.459184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698292.459193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f5187388000 x1874531146052992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698292.459204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146052992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.459211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.459214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.459226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146052992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698292.459232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146053120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698292.459248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698292.459259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f518738a680 x1874531146053120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698292.459269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146053120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.459276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.459279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.459281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146053120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698292.459287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146053248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698292.459305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698292.459316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698292/real 1787698292] req@ffff8f5187388a80 x1874531146053248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698292.459328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146053248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698292.459336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698292.459339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698292.459342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146053248:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698293.354494:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698294.378552:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698295.402504:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698296.426521:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698297.450495:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698297.578440:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698297.578450:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698297.578454:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698297.578464:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578474:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578479:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698297.578511:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146053376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698297.578539:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698297.578542:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698297.578544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698297.578548:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578552:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578556:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698297.578564:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698297.578575:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146053504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698297.578583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146053376:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698297.578585:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698297.578591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698297.578593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698297.578596:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578599:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578601:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698297.578604:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698297.578613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146053632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698297.578649:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698297.578653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698297.578654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698297.578657:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578660:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698297.578665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698297.578665:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698297.578671:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698297.578682:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146053760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698297.578694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698297.578696:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698297.578697:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698297.578701:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00100000:0.0:1787698297.578704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f518738aa00 x1874531146053376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698297.578704:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578711:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698297.578717:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698297.578729:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146053888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698297.578735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146053376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698297.578745:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698297.578748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698297.578750:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698297.578753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698297.578755:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698297.578757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146053376:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698297.578763:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578767:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578771:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698297.578775:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698297.578785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698297.578790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146054016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698297.578802:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698297.578807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698297.578809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698297.578813:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698297.578818:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00100000:0.0:1787698297.578820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f5187389500 x1874531146053504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698297.578823:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578831:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698297.578836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146053504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698297.578839:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698297.578842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.578845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.578848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698297.578855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698297.578857:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146054144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698297.578867:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698297.578870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698297.578874:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698297.578877:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698297.578881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f518738a300 x1874531146053632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698297.578884:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578892:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698297.578896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146053632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698297.578900:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698297.578909:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698297.578911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.578916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.578918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698297.578925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698297.578926:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146054272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698297.578937:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698297.578938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698297.578943:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698297.578945:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698297.578948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f5187389c00 x1874531146053760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698297.578949:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6197 00000100:00080000:3.0:1787698297.578955:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698297.578961:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698297.578963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146053760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698297.578967:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698297.578974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.578977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.578980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698297.578982:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146054400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698297.578987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698297.579006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698297.579013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f518738bb80 x1874531146053888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698297.579020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146053888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698297.579025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.579027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.579029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146053888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698297.579038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698297.579048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698297.579053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f5187388380 x1874531146054016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698297.579059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146054016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698297.579064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.579066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.579067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698297.579070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698297.579078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698297.579083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f518738b800 x1874531146054144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698297.579089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146054144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698297.579093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.579095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.579097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698297.579099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698297.579107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698297.579112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f518738ad80 x1874531146054272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698297.579118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146054272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698297.579123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.579124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.579125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698297.579134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698297.579144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698297.579149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698297/real 1787698297] req@ffff8f518738aa00 x1874531146054400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698297.579155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146054400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698297.579159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698297.579161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698297.579163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146054400:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698298.474521:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698299.498464:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698300.522524:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698301.546521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698302.570509:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698302.698408:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698419:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698302.698423:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698302.698441:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698455:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698462:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698302.698496:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146054528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698302.698524:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698527:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698302.698529:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698302.698534:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698552:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698562:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698302.698571:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698302.698576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146054528:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698302.698597:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146054656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698302.698608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698612:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698302.698613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698302.698617:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698621:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698302.698630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698302.698644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146054784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698302.698650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698302.698658:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698662:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698302.698664:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698302.698669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698673:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698677:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698302.698683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698302.698688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f5187388a80 x1874531146054528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698302.698697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146054912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698302.698709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146054528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698302.698709:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698302.698718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698302.698725:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:0.0:1787698302.698726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.698731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698302.698731:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698302.698738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146054528:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698302.698741:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698302.698747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698302.698759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146054656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698302.698761:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146055040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698302.698778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698302.698779:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698784:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698302.698786:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698302.698789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f518738a680 x1874531146054656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698302.698793:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00100000:0.0:1787698302.698803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146054656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698302.698810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698302.698810:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698302.698821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698302.698822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698302.698831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146054656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698302.698837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146055168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698302.698843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146054784:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698302.698849:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698852:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698302.698853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698302.698856:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698859:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698862:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698302.698866:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698302.698871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698302.698883:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146055296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698302.698884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f5187388000 x1874531146054784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698302.698898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146054784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698302.698906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698914:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698302.698918:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698302.698920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698302.698923:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698927:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.698933:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698302.698936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698302.698938:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698302.698943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146054784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698302.698951:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146055424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698302.698954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146054912:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698302.698969:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698302.698972:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698302.698974:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698302.698988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698302.698992:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6202 00000100:00080000:3.0:1787698302.698997:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698302.699004:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698302.699005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f518738b480 x1874531146054912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698302.699010:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698302.699025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146054912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698302.699027:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146055552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698302.699050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.699054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698302.699058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146054912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698302.699073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146055040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698302.699092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698302.699102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f5187389f80 x1874531146055040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698302.699113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146055040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698302.699119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.699122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698302.699125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146055040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698302.699131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698302.699146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698302.699153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f5187388e00 x1874531146055168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698302.699159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146055168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698302.699165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.699166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698302.699168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698302.699171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698302.699180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698302.699185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f518738b100 x1874531146055296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698302.699191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146055296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698302.699205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.699207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698302.699208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698302.699212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698302.699222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698302.699227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f5187389180 x1874531146055424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698302.699234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146055424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698302.699238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.699239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698302.699241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698302.699244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698302.699252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698302.699256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698302/real 1787698302] req@ffff8f5187388a80 x1874531146055552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698302.699262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146055552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698302.699266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698302.699268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698302.699269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146055552:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698303.594548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698304.618603:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698305.642546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698306.666530:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698307.690572:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698307.818524:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818534:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698307.818538:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818551:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818565:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818573:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698307.818622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146055680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698307.818651:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818654:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698307.818656:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818659:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818662:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818666:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818676:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698307.818688:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146055808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698307.818700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698307.818705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698307.818707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146055680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698307.818710:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818714:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818718:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818721:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698307.818729:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146055936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698307.818735:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818738:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698307.818739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818741:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818743:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818745:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698307.818755:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146056064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698307.818760:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698307.818763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818766:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818770:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818772:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818775:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698307.818784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146056192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698307.818787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698307.818791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818796:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698307.818797:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818800:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818802:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818804:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698307.818813:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146056320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698307.818819:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698307.818822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818824:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818827:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698307.818829:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818833:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698307.818842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704ed80 x1874531146055680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698307.818847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146056448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698307.818880:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818884:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698307.818888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146055680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698307.818888:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698307.818898:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:0.0:1787698307.818903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698307.818905:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698307.818909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.818915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146055680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698307.818915:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818924:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698307.818940:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146056576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698307.818942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146055808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698307.818958:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698307.818961:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698307.818964:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698307.818967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698307.818973:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6207 00000100:00080000:2.0:1787698307.818979:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698307.818980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704dc00 x1874531146055808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698307.818987:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698307.818997:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698307.819001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146055808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698307.819014:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146056704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698307.819021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146055808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698307.819049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146055936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698307.819072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704ea00 x1874531146055936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146055936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146055936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698307.819128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146056064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698307.819147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704e300 x1874531146056064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146056064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146056064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698307.819186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146056192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698307.819217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704d880 x1874531146056192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146056192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146056192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698307.819259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698307.819276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704d500 x1874531146056320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146056320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698307.819315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698307.819340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704f480 x1874531146056448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146056448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698307.819425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698307.819444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704c700 x1874531146056576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146056576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698307.819501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698307.819519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698307.819529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698307/real 1787698307] req@ffff8f518704e680 x1874531146056704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698307.819540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146056704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698307.819547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698307.819550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698307.819553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146056704:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698308.714541:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698309.738578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698310.762497:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698311.786558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698312.810537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698312.938490:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938500:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698312.938504:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698312.938513:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938523:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938531:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698312.938567:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146056832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938597:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698312.938599:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698312.938603:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938607:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938611:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698312.938621:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698312.938632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146056832:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698312.938643:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146056960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938655:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698312.938660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698312.938663:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938681:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938685:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698312.938690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787698312.938690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698312.938705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146057088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938714:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938717:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698312.938719:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698312.938723:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938728:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938732:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698312.938735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183644700 x1874531146056832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698312.938738:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698312.938755:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146057216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938781:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938785:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698312.938787:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698312.938792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146056832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.938795:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938800:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938807:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698312.938812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698312.938813:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698312.938819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.938823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146056832:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698312.938837:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146057344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938847:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938851:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787698312.938853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146056960:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698312.938854:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698312.938863:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938868:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938872:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698312.938877:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698312.938888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698312.938891:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146057472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938903:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698312.938904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183644000 x1874531146056960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.938913:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698312.938917:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698312.938924:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00100000:0.0:1787698312.938928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146056960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.938929:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.938938:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698312.938942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698312.938945:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698312.938948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.938953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146056960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698312.938962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146057088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698312.938962:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146057600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698312.938977:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.938981:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698312.938983:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698312.938985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698312.938991:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.938996:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698312.938999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183646680 x1874531146057088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.939003:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698312.939013:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698312.939019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146057088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.939028:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146057728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698312.939036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146057088:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698312.939053:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698312.939057:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698312.939061:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698312.939065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146057216:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787698312.939068:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6212 00000100:00080000:1.0:1787698312.939075:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698312.939079:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698312.939084:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698312.939090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698312.939100:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146057856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698312.939104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183647800 x1874531146057216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698312.939128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146057216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698312.939136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146057216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698312.939147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146057344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698312.939162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698312.939170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183647480 x1874531146057344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698312.939180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146057344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698312.939187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146057344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698312.939211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698312.939227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698312.939235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183645c00 x1874531146057472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698312.939245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146057472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698312.939253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698312.939265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698312.939277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698312.939285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183645180 x1874531146057600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698312.939296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146057600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698312.939302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698312.939313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698312.939326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698312.939334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183647100 x1874531146057728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698312.939343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146057728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698312.939351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698312.939402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698312.939418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698312.939427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff8f5183644380 x1874531146057856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698312.939438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146057856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698312.939445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698312.939448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698312.939451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146057856:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698313.834716:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698314.858595:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698315.882530:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698316.906574:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698317.930550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698318.058467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058474:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698318.058477:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058484:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058498:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698318.058526:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146057984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698318.058548:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058551:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698318.058552:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058555:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058557:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058559:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058566:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698318.058575:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146058112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698318.058582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698318.058586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058588:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058590:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058592:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058594:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698318.058605:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146058240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698318.058606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146057984:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698318.058627:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058629:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698318.058630:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058633:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058636:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058638:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058643:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698318.058651:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146058368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698318.058657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698318.058660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058663:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058666:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058667:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698318.058675:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146058496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698318.058680:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058682:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698318.058684:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058686:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058689:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698318.058696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698318.058697:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058703:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698318.058710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146058624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698318.058721:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058723:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698318.058725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058728:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058730:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058731:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698318.058743:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146058752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698318.058750:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058752:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698318.058753:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698318.058756:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00100000:0.0:1787698318.058761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f518738aa00 x1874531146057984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698318.058761:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698318.058811:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146058880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698318.058812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146057984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.058841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.058851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.058864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146057984:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698318.058867:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698318.058871:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698318.058873:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698318.058874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698318.058882:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6217 00000100:00080000:3.0:1787698318.058888:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698318.058892:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698318.058896:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698318.058907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698318.058912:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146059008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698318.058921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f518738ad80 x1874531146058112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.058932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146058112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.058938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.058949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.058952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698318.058980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698318.059002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f518738b800 x1874531146058240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146058240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698318.059038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698318.059052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f5187388380 x1874531146058368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146058368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698318.059075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698318.059085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f518738bb80 x1874531146058496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146058496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146058496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698318.059119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146058624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698318.059138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f5187389c00 x1874531146058624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146058624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146058624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698318.059169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146058752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698318.059181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f518738a300 x1874531146058752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146058752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146058752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698318.059204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146058880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698318.059213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f5187389500 x1874531146058880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146058880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146058880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698318.059237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146059008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698318.059253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698318.059263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698318/real 1787698318] req@ffff8f5187388a80 x1874531146059008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698318.059272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146059008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698318.059295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698318.059298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698318.059301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146059008:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698318.954512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698319.978597:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698321.002549:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698322.026564:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698323.051913:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698323.178467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698323.178481:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698323.178492:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178502:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.178509:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698323.178543:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146059136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698323.178572:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178576:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698323.178578:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698323.178582:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178585:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.178589:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.178598:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698323.178612:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146059264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698323.178623:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178627:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698323.178630:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698323.178632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146059136:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698323.178636:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.178660:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.178665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698323.178677:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146059392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698323.178687:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178689:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698323.178692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698323.178696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178699:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.178703:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.178708:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698323.178717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698323.178720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146059520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698323.178731:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178735:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698323.178737:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698323.178740:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178744:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.178747:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.178753:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698323.178762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f5187389180 x1874531146059136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698323.178766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146059648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698323.178790:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178796:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698323.178798:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698323.178805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146059136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698323.178805:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178809:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.178817:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698323.178822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698323.178824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698323.178829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.178834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146059136:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698323.178840:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146059776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698323.178851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698323.178859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059264:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698323.178866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698323.178870:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698323.178876:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:3.0:1787698323.178880:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698323.178885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698323.178887:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.178894:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698323.178901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f518738b100 x1874531146059264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698323.178908:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146059904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698323.178921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146059264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698323.178925:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.178933:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698323.178937:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698323.178939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698323.178942:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00080000:0.0:1787698323.178947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698323.178949:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698323.178953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059264:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698323.178957:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.178964:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698323.178965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698323.178977:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146060032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698323.178992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698323.178995:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698323.179000:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698323.179002:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698323.179007:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6222 00000100:00100000:0.0:1787698323.179008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f5187388e00 x1874531146059392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698323.179011:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698323.179019:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698323.179026:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698323.179030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146059392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698323.179039:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146060160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698323.179067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698323.179083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698323.179104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698323.179116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f5187389f80 x1874531146059520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698323.179127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146059520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698323.179135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698323.179147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698323.179167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698323.179177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f518738b480 x1874531146059648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698323.179207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146059648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698323.179216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146059648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698323.179230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146059776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698323.179250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698323.179273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f5187388000 x1874531146059776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698323.179285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146059776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698323.179292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146059776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698323.179316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146059904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698323.179335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698323.179345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f518738a680 x1874531146059904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698323.179405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146059904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698323.179414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146059904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698323.179426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146060032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698323.179447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698323.179459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f5187389180 x1874531146060032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698323.179470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146060032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698323.179477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146060032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698323.179489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146060160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698323.179506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698323.179528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698323/real 1787698323] req@ffff8f5187388a80 x1874531146060160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698323.179539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146060160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698323.179547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698323.179550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698323.179553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146060160:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698324.074546:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698325.098586:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698326.122649:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698327.146604:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698328.170527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698328.298433:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298451:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698328.298456:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298466:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298476:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298492:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698328.298528:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146060288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698328.298555:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698328.298560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298566:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298569:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698328.298577:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698328.298587:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146060416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698328.298594:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298596:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698328.298597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298600:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298601:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298605:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698328.298607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146060288:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698328.298609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698328.298619:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146060544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698328.298625:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298627:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698328.298628:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298630:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298632:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298634:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698328.298636:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698328.298644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146060672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698328.298649:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298651:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698328.298651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298653:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298655:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298657:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698328.298659:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698328.298665:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146060800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698328.298672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698328.298672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298678:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698328.298680:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298685:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298688:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298693:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698328.298697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698328.298709:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146060928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698328.298715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f5187389500 x1874531146060288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698328.298727:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298733:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698328.298736:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698328.298744:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298749:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698328.298754:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698328.298756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146060288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698328.298760:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698328.298774:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146061056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698328.298788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698328.298790:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698328.298795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698328.298795:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698328.298799:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698328.298803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146060288:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698328.298807:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298814:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698328.298817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060416:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698328.298821:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698328.298828:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698328.298841:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146061184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698328.298849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698328.298857:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698328.298861:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698328.298863:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698328.298867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f518738a300 x1874531146060416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698328.298870:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6227 00000100:00080000:3.0:1787698328.298877:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698328.298888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146060416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.298896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.298899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698328.298900:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698328.298904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060416:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698328.298908:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698328.298923:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146061312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698328.298931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698328.298950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.298960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f5187389c00 x1874531146060544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.298971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146060544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.298984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.298986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.298989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698328.298995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698328.299009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.299017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f518738bb80 x1874531146060672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.299026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146060672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.299034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.299037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.299040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698328.299045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698328.299216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.299228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f5187388380 x1874531146060800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.299239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146060800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.299247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.299250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.299253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146060800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698328.299259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146060928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698328.299273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.299282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f518738b800 x1874531146060928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.299291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146060928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.299300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.299302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.299305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146060928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698328.299310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146061056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698328.299324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.299332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f518738ad80 x1874531146061056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.299342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146061056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.299349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.299352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.299382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146061056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698328.299389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146061184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698328.299406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.299417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f518738aa00 x1874531146061184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.299427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146061184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.299446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.299449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.299453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146061184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698328.299470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146061312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698328.299492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698328.299501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698328/real 1787698328] req@ffff8f5187388a80 x1874531146061312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698328.299512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146061312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698328.299520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698328.299522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698328.299525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146061312:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698329.194531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698330.218628:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698331.242526:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698332.266571:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698333.290558:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698333.418435:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418453:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698333.418456:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698333.418466:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418476:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418482:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698333.418516:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146061440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698333.418546:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418549:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698333.418551:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698333.418555:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418559:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418563:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418572:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698333.418584:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146061568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698333.418588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146061440:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698333.418601:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418605:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698333.418606:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698333.418611:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418614:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418617:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418622:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698333.418634:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146061696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698333.418644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698333.418644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698333.418652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698333.418657:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418661:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418665:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418669:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698333.418684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f5187389180 x1874531146061440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698333.418685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146061824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698333.418714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418718:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698333.418719:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698333.418723:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418729:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418735:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698333.418745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146061440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698333.418752:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146061952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698333.418759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698333.418765:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698333.418768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698333.418770:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698333.418774:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698333.418775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146061440:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698333.418783:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418787:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418791:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698333.418803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698333.418809:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146062080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698333.418820:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418824:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698333.418826:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698333.418830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698333.418833:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:3.0:1787698333.418838:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418842:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418847:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698333.418848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f518738a680 x1874531146061568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.418863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146061568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698333.418871:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146062208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698333.418874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698333.418885:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418888:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698333.418890:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698333.418895:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00080000:0.0:1787698333.418898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698333.418898:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698333.418908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698333.418908:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418916:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698333.418920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698333.418930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146062336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698333.418940:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698333.418946:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698333.418947:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698333.418950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698333.418953:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6233 00000100:00100000:0.0:1787698333.418964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f5187388000 x1874531146061696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698333.418966:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698333.418974:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698333.418982:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698333.418988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146061696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698333.418996:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146062464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698333.419036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698333.419053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698333.419074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698333.419085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f518738b480 x1874531146061824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.419096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146061824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698333.419103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698333.419115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698333.419133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698333.419143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f5187389f80 x1874531146061952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.419154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146061952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698333.419161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146061952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698333.419183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698333.419200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698333.419210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f5187388e00 x1874531146062080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.419221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146062080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698333.419228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698333.419239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698333.419256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698333.419265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f518738b100 x1874531146062208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.419276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146062208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698333.419282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698333.419293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698333.419306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698333.419313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f5187389180 x1874531146062336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.419324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146062336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698333.419331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698333.419340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698333.419379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698333.419391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698333/real 1787698333] req@ffff8f5187388a80 x1874531146062464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698333.419402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146062464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698333.419409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698333.419412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698333.419414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146062464:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698334.314530:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698335.338548:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698336.362506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698337.386579:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698338.410497:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698338.538498:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538508:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698338.538512:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.538525:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538536:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.538543:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698338.538586:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146062592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698338.538619:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538623:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698338.538625:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.538630:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538634:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.538638:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698338.538648:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698338.538659:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146062720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698338.538672:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698338.538676:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.538679:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538683:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.538689:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698338.538691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146062592:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787698338.538693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698338.538704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146062848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698338.538710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538712:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698338.538714:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.538717:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538718:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.538721:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698338.538724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698338.538732:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146062976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698338.538737:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698338.538740:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.538744:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538767:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.538770:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698338.538773:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698338.538780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698338.538785:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146063104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698338.538794:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538797:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698338.538799:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.538802:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.538817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698338.538822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698338.538828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183647b80 x1874531146062592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698338.538845:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146063232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698338.538849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146062592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698338.538894:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.538897:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698338.538898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698338.538900:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698338.538906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698338.538910:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.538918:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698338.538919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146062592:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698338.538957:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698338.538960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146062720:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787698338.538965:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698338.538986:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146063360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698338.538995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698338.539009:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698338.539011:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698338.539012:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.539019:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00100000:0.0:1787698338.539024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183646a00 x1874531146062720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698338.539024:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.539036:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698338.539047:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698338.539057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146062720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698338.539066:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146063488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698338.539069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146062720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698338.539097:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698338.539098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146062848:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698338.539105:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698338.539108:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698338.539112:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6238 00000100:00080000:1.0:1787698338.539117:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698338.539120:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698338.539124:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698338.539135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698338.539135:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146063616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698338.539153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183646d80 x1874531146062848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146062848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146062848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698338.539182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146062976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698338.539193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698338.539198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183644e00 x1874531146062976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146062976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146062976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698338.539216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146063104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698338.539224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698338.539229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183644380 x1874531146063104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146063104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146063104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698338.539246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698338.539255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698338.539260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183647100 x1874531146063232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146063232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698338.539287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698338.539296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698338.539302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183645180 x1874531146063360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146063360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698338.539328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698338.539345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698338.539383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183645c00 x1874531146063488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146063488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698338.539404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698338.539417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698338.539425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698338/real 1787698338] req@ffff8f5183647480 x1874531146063616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698338.539435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146063616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698338.539443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698338.539446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698338.539449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146063616:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698339.434511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698340.458574:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698341.482651:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698342.506542:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698343.530546:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698343.658455:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658463:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698343.658467:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658477:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658487:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698343.658494:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698343.658531:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146063744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698343.658555:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698343.658560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658565:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698343.658572:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698343.658580:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698343.658593:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146063872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698343.658602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146063744:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698343.658605:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658611:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698343.658613:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658617:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658621:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698343.658625:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698343.658630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698343.658642:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146064000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698343.658656:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658660:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698343.658663:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698343.658664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698343.658668:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658673:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698343.658677:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698343.658682:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698343.658694:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146064128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698343.658702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698343.658707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704c000 x1874531146063744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698343.658708:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698343.658712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658720:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658723:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698343.658728:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698343.658735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698343.658749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146064256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698343.658755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146063744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698343.658763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658769:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698343.658770:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698343.658776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698343.658776:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658783:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698343.658784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698343.658789:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698343.658792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146063744:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787698343.658796:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698343.658809:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146064384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698343.658818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146063872:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698343.658821:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658826:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698343.658828:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658832:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658837:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698343.658862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698343.658868:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698343.658875:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698343.658878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704c380 x1874531146063872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698343.658889:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146064512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698343.658894:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146063872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698343.658905:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658911:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698343.658912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698343.658914:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658920:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00080000:2.0:1787698343.658924:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698343.658929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698343.658932:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698343.658937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146063872:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698343.658940:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698343.658946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146064000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698343.658953:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146064640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698343.658970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698343.658971:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698343.658975:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698343.658978:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698343.658982:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6243 00000100:00100000:0.0:1787698343.658986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704ce00 x1874531146064000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698343.658988:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698343.658996:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698343.659005:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698343.659009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146064000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698343.659019:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146064768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698343.659036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146064000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698343.659053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146064128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698343.659070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698343.659079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704f800 x1874531146064128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698343.659089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146064128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698343.659096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146064128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698343.659107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146064256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698343.659122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698343.659131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704df80 x1874531146064256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698343.659142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146064256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698343.659161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146064256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698343.659184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698343.659201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698343.659211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704fb80 x1874531146064384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698343.659221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146064384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698343.659229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698343.659239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698343.659254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698343.659262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704d180 x1874531146064512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698343.659272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146064512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698343.659279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698343.659289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698343.659303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698343.659320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704c000 x1874531146064640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698343.659330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146064640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698343.659338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698343.659349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698343.659397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698343.659405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698343/real 1787698343] req@ffff8f518704e680 x1874531146064768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698343.659412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146064768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698343.659417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698343.659419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698343.659420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146064768:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698344.555199:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698345.578514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698346.602596:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698347.626863:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698348.650557:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698348.778494:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698348.778504:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698348.778508:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778517:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778527:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.778535:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698348.778569:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146064896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778599:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698348.778602:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698348.778604:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778609:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778612:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.778616:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778625:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698348.778640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146065024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778651:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698348.778652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146064896:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698348.778658:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698348.778663:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778667:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778671:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.778675:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698348.778693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146065152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698348.778706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698348.778708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778712:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778715:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.778719:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698348.778737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146065280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778746:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698348.778750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698348.778766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698348.778771:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778779:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778783:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.778788:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698348.778798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698348.778811:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146065408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778822:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698348.778825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065280:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698348.778827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698348.778840:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778847:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778851:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698348.778853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065408:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698348.778859:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698348.778881:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146065536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778894:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698348.778898:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698348.778900:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698348.778905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065536:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698348.778906:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778913:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.778917:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778923:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698348.778936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146065664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.778947:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698348.778950:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698348.778952:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.778957:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.778961:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698348.778968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065664:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698348.778968:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.778978:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698348.778992:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146065792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698348.779002:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698348.779006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065792:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698348.779007:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698348.779012:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698348.779018:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6248 00000100:00080000:3.0:1787698348.779023:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698348.779027:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698348.779032:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698348.779045:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146065920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698348.779089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065920:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698349.674501:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698350.698551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698351.722574:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698351.850748:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787698351.850807:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698351.850830:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850896:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850912:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850921:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850929:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850936:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850944:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850953:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698351.850960:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698351.851068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f5187388a80 x1874531146065920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146065920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698351.851131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f5187389500 x1874531146065792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146065792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698351.851162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f518738a300 x1874531146065664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146065664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698351.851191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f5187389c00 x1874531146065536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146065536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146065536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698351.851221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f518738bb80 x1874531146065408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146065408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698351.851253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f518738ad80 x1874531146065024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146065024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698351.851286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f518738b800 x1874531146065152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146065152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698351.851316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f5187388380 x1874531146065280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146065280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146065280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698351.851345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698348/real 1787698351] req@ffff8f518738aa00 x1874531146064896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698351.851525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146064896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698351.851534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698351.851538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698351.851541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146064896:192.168.201.125@tcp:250:kworker.0 00010000:00010000:3.0:1787698352.746582:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698353.770579:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698353.898606:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698353.898624:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898638:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898652:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898660:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698353.898716:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146066048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898754:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698353.898760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898765:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898768:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898772:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.898784:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698353.898798:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146066176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898807:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898810:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698353.898812:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898815:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898818:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898822:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.898826:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698353.898835:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146066304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898843:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698353.898847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898850:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898853:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898856:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.898860:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698353.898870:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146066432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898880:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698353.898882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898885:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898888:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898891:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.898895:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698353.898906:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146066560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898913:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898917:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698353.898918:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898923:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898926:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898930:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.898934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698353.898944:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146066688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898953:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898956:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698353.898958:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898961:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.898965:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.898968:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.898971:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698353.898980:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146066816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.898989:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.898992:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698353.898993:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.898997:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.899000:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.899002:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.899006:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698353.899014:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146066944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.899021:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698353.899024:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698353.899026:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698353.899029:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6253 00000100:00080000:1.0:1787698353.899032:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698353.899035:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698353.899039:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698353.899049:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146067072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698353.899155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146066048:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787698353.899236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183647800 x1874531146066048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146066048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146066048:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698353.899328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698353.899348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183646680 x1874531146066176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146066176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698353.899438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698353.899458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183644000 x1874531146066304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146066304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698353.899498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698353.899514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183644700 x1874531146066432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146066432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698353.899558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698353.899572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183647480 x1874531146066560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146066560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146066560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698353.899610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146066688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698353.899625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183645c00 x1874531146066688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146066688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146066688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698353.899678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146066816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698353.899727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183645180 x1874531146066816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146066816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146066816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698353.899768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146066944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698353.899786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183647100 x1874531146066944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146066944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146066944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698353.899824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146067072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698353.899838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698353.899847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff8f5183644380 x1874531146067072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698353.899858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146067072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698353.899865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698353.899868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698353.899871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146067072:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698354.794550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698355.818512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698356.842534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698357.866466:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698358.890516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698359.018487:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698359.018494:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698359.018496:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018503:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018511:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018516:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698359.018544:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146067200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698359.018564:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698359.018566:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698359.018567:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018570:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018572:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018575:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018593:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698359.018607:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146067328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698359.018618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698359.018622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698359.018624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146067200:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698359.018625:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018632:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018640:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018644:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698359.018656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146067456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698359.018665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698359.018668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698359.018670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018674:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018677:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018680:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698359.018692:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146067584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698359.018698:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698359.018703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698359.018705:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698359.018707:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018710:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018713:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018715:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018718:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698359.018727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146067712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698359.018736:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698359.018741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704c700 x1874531146067200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698359.018741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698359.018744:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018753:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018758:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018764:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018768:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698359.018769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146067200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698359.018779:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146067840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698359.018782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698359.018789:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698359.018792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698359.018793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698359.018797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698359.018800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146067200:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698359.018802:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018807:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018810:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018813:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698359.018819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146067968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698359.018826:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698359.018830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698359.018831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698359.018839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018843:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018847:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698359.018850:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018855:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698359.018859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698359.018868:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146068096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698359.018874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704f480 x1874531146067328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698359.018876:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698359.018884:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698359.018888:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698359.018897:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6258 00000100:00080000:2.0:1787698359.018903:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698359.018908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146067328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698359.018909:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698359.018917:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698359.018922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.018927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698359.018930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146068224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698359.018935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698359.018944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698359.018962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.018970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704d500 x1874531146067456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.018977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146067456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.018981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.018983:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.018985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698359.018988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698359.018998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.019003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704d880 x1874531146067584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.019013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146067584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.019025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.019037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.019041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698359.019047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698359.019065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.019076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704e300 x1874531146067712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.019087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146067712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.019094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.019096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.019099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146067712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698359.019105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146067840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698359.019123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.019133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704ea00 x1874531146067840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.019143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146067840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.019150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.019153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.019156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146067840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698359.019174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146067968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698359.019191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.019198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704dc00 x1874531146067968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.019205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146067968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.019210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.019212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.019213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146067968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698359.019217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146068096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698359.019226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.019231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704ed80 x1874531146068096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.019246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146068096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.019251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.019253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.019255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146068096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698359.019258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146068224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698359.019267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698359.019273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698359/real 1787698359] req@ffff8f518704c700 x1874531146068224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698359.019279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146068224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698359.019283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698359.019285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698359.019286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146068224:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698359.914522:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698360.938538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698361.962553:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698362.986536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698364.010589:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698364.138523:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698364.138536:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698364.138541:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.138555:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138569:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698364.138576:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698364.138638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146068352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698364.138671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698364.138675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698364.138677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.138682:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138686:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698364.138690:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698364.138701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698364.138717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146068480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698364.138722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146068352:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698364.138736:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698364.138740:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698364.138742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.138747:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138751:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698364.138754:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698364.138759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698364.138771:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146068608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698364.138786:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698364.138791:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698364.138792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698364.138795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.138801:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138805:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698364.138809:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698364.138814:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698364.138826:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146068736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698364.138835:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698364.138840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704e680 x1874531146068352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698364.138841:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698364.138845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.138854:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138858:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698364.138860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146068352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698364.138867:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698364.138873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698364.138874:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698364.138879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.138883:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146068352:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698364.138890:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146068864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698364.138902:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698364.138906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698364.138907:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698364.138912:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.138918:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138923:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698364.138927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698364.138929:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698364.138935:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698364.138939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704c000 x1874531146068480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698364.138950:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146068992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698364.138957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146068480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698364.138969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698364.138969:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698364.138976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698364.138977:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698364.138980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698364.138981:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698364.138988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698364.138990:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.138995:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698364.139000:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698364.139006:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698364.139015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698364.139021:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146069120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698364.139025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704d180 x1874531146068608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698364.139044:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698364.139048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146068608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698364.139050:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698364.139054:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698364.139059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698364.139063:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:0.0:1787698364.139065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698364.139070:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698364.139076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068736:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698364.139079:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698364.139087:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698364.139093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698364.139104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704fb80 x1874531146068736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698364.139104:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146069248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698364.139138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146068736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698364.139143:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698364.139149:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698364.139152:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698364.139158:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6263 00000100:00080000:2.0:1787698364.139164:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698364.139171:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698364.139174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698364.139177:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698364.139181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698364.139194:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146069376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698364.139196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698364.139226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698364.139240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704df80 x1874531146068864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698364.139251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146068864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698364.139257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698364.139259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146068864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698364.139273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146068992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698364.139287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698364.139294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704f800 x1874531146068992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698364.139304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146068992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698364.139313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698364.139316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146068992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698364.139326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146069120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698364.139344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698364.139390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704ce00 x1874531146069120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698364.139403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146069120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698364.139412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698364.139415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146069120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698364.139446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146069248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698364.139466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698364.139475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704c380 x1874531146069248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698364.139482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146069248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698364.139487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698364.139500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146069248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698364.139507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146069376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698364.139520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698364.139528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698364/real 1787698364] req@ffff8f518704e680 x1874531146069376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698364.139534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146069376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698364.139540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698364.139542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698364.139544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146069376:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698365.034510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698366.058526:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698367.082516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698368.106543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698369.130570:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698369.258469:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258481:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698369.258484:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258496:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00080000:3.0:1787698369.258509:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258517:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698369.258560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146069504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698369.258584:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698369.258588:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258592:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00080000:3.0:1787698369.258594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698369.258608:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698369.258618:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146069632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698369.258624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146069504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698369.258629:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258633:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698369.258635:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258639:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00080000:3.0:1787698369.258644:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258648:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698369.258653:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698369.258668:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146069760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698369.258678:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258682:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698369.258684:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258688:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6268 00000800:00000100:0.0:1787698369.258692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698369.258695:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258701:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698369.258707:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698369.258720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146069888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698369.258727:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698369.258731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698369.258733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f5187389180 x1874531146069504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698369.258737:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00080000:3.0:1787698369.258744:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258750:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698369.258756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146069504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698369.258756:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698369.258772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698369.258773:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146070016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698369.258780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.258789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146069504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698369.258790:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258797:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698369.258800:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258805:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00080000:3.0:1787698369.258808:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258811:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698369.258815:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698369.258821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146069632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698369.258826:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146070144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698369.258834:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258836:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698369.258838:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258840:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6268 00000800:00000100:0.0:1787698369.258844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698369.258845:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258850:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698369.258854:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698369.258859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f518738b100 x1874531146069632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698369.258865:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146070272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698369.258887:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698369.258892:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698369.258895:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698369.258899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146069632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698369.258901:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00080000:3.0:1787698369.258908:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698369.258915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698369.258916:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698369.258923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698369.258925:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698369.258927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146069632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698369.258935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146069760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698369.258943:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146070400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698369.258952:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698369.258953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698369.258958:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698369.258960:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698369.258965:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6268 00000100:00100000:0.0:1787698369.258966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f5187388e00 x1874531146069760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698369.258968:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698369.258975:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698369.258980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146069760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698369.258981:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698369.258992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.258997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146069760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698369.259007:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146070528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698369.259010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146069888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698369.259038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698369.259050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f5187389f80 x1874531146069888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698369.259061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146069888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698369.259084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.259087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146069888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698369.259096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146070016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698369.259115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698369.259126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f518738b480 x1874531146070016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698369.259137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146070016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698369.259145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.259148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146070016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698369.259169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698369.259187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698369.259196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f5187388000 x1874531146070144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698369.259207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146070144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698369.259216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.259219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698369.259228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698369.259245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698369.259255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f518738a680 x1874531146070272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698369.259266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146070272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698369.259273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.259276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698369.259285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698369.259301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698369.259310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f5187389180 x1874531146070400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698369.259321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146070400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698369.259326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.259328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698369.259334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698369.259345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698369.259395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698369/real 1787698369] req@ffff8f518738aa00 x1874531146070528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698369.259419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146070528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698369.259428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698369.259432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698369.259436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146070528:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698370.154614:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698371.178564:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698372.202517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698373.226487:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698374.251469:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698374.378478:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378485:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698374.378487:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378494:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378503:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378508:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698374.378536:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146070656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378557:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698374.378558:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378561:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378563:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378565:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378571:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698374.378578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146070784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378585:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698374.378588:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378592:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378594:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378597:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698374.378602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146070656:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698374.378607:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146070912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378616:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698374.378618:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378620:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378622:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378624:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698374.378633:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146071040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378640:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698374.378641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378644:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378659:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698374.378667:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146071168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378674:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698374.378677:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698374.378679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698374.378682:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378685:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378687:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698374.378696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146071296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378704:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698374.378705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378707:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378709:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378711:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378713:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698374.378720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146071424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698374.378727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183644e00 x1874531146070656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698374.378727:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378732:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698374.378734:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698374.378737:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378738:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378740:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378743:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698374.378751:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146071552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698374.378758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698374.378760:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698374.378762:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698374.378764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146070656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698374.378766:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6274 00000100:00080000:1.0:1787698374.378770:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698374.378774:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698374.378779:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698374.378784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698374.378788:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146071680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698374.378792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.378800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146070656:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698374.378808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146070784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698374.378841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.378854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183646d80 x1874531146070784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.378872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146070784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.378880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.378883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.378886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146070784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698374.378911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146070912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698374.378930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.378939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183646a00 x1874531146070912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.378951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146070912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.378959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.378962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.378965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146070912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698374.378970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146071040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698374.378988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.378998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183647b80 x1874531146071040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.379009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146071040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.379020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.379024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.379027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146071040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698374.379032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146071168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698374.379059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.379069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183644380 x1874531146071168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.379080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146071168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.379087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.379090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.379093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146071168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698374.379098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698374.379115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.379125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183647100 x1874531146071296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.379135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146071296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.379142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.379145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.379149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698374.379154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698374.379171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.379180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183645180 x1874531146071424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.379190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146071424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.379197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.379199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.379202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698374.379208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698374.379224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.379234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183645c00 x1874531146071552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.379244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146071552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.379251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.379254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.379257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698374.379262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698374.379278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698374.379288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698374/real 1787698374] req@ffff8f5183647480 x1874531146071680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698374.379298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146071680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698374.379305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698374.379308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698374.379311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146071680:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698375.274492:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698376.298503:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698377.322526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698378.346516:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698379.370524:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698379.498477:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498485:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698379.498488:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498496:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498505:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498511:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698379.498543:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146071808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498565:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498569:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698379.498571:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498576:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498580:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498584:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498595:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698379.498612:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146071936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498623:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498625:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698379.498627:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698379.498632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146071808:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698379.498632:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498638:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498642:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698379.498656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146072064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498663:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698379.498667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498671:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498674:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498677:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698379.498684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146072192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698379.498694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498696:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698379.498704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698379.498705:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498712:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698379.498719:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146072320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498726:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498728:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698379.498730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498732:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498734:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498746:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498749:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698379.498756:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146072448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498762:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698379.498765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704c700 x1874531146071808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698379.498765:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698379.498769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498782:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498785:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698379.498788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146071808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698379.498790:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698379.498804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698379.498806:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146072576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698379.498810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.498817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146071808:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698379.498821:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498825:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698379.498826:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498830:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498832:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498834:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498838:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698379.498842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146071936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698379.498847:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146072704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698379.498856:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698379.498858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698379.498859:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698379.498862:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6279 00000100:00080000:2.0:1787698379.498866:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698379.498869:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698379.498872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698379.498873:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698379.498884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146072832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698379.498888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704ed80 x1874531146071936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.498913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146071936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.498919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.498922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.498925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146071936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698379.498929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146072064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698379.498942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.498948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704dc00 x1874531146072064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.498956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146072064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.498964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.498967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.498971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146072064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698379.498978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146072192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698379.498998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.499010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704ea00 x1874531146072192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.499020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146072192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.499030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.499032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.499035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146072192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698379.499041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146072320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698379.499061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.499072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704e300 x1874531146072320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.499085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146072320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.499094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.499097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.499101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146072320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698379.499105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698379.499120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.499128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704d880 x1874531146072448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.499136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146072448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.499142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.499144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.499146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698379.499150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698379.499160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.499165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704d500 x1874531146072576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.499172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146072576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.499177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.499179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.499181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698379.499195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698379.499214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.499221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704f480 x1874531146072704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.499256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146072704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.499262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.499264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.499266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698379.499270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698379.499284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698379.499291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698379/real 1787698379] req@ffff8f518704c700 x1874531146072832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698379.499299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146072832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698379.499304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698379.499306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698379.499307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146072832:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698380.394569:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698381.418519:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698382.442509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698383.466604:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698384.490616:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698384.618498:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618509:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698384.618514:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.618526:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618540:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618548:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698384.618591:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146072960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698384.618615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698384.618619:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.618622:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618626:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618630:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.618643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698384.618661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146072960:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698384.618664:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146073088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698384.618685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698384.618690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.618696:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618700:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618705:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.618710:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698384.618722:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146073216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698384.618734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698384.618738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698384.618747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698384.618747:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618756:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618761:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.618765:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698384.618775:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146073344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698384.618782:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698384.618786:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.618792:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618802:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618805:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.618809:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698384.618817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704e680 x1874531146072960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698384.618818:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146073472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698384.618847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698384.618863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.618876:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618881:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618885:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.618889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698384.618890:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146072960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698384.618904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146073600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698384.618908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698384.618922:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698384.618924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698384.618928:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698384.618931:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.618934:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618936:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618943:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698384.618944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146072960:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787698384.618949:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698384.618960:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146073728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698384.618966:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698384.618969:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698384.618971:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698384.618978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073088:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698384.618978:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.618987:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.618992:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.618997:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698384.619011:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146073856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698384.619020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698384.619032:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698384.619034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704c380 x1874531146073088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698384.619036:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698384.619043:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698384.619060:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6284 00000100:00080000:2.0:1787698384.619068:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698384.619079:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698384.619092:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698384.619097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146073088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698384.619109:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146073984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698384.619125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698384.619162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698384.619188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704ce00 x1874531146073216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146073216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698384.619221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698384.619234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704f800 x1874531146073344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146073344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698384.619264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698384.619274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704df80 x1874531146073472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146073472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146073472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698384.619299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698384.619310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704fb80 x1874531146073600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146073600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698384.619337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698384.619347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704d180 x1874531146073728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146073728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698384.619412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698384.619428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704c000 x1874531146073856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146073856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698384.619461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698384.619471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698384.619477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698384/real 1787698384] req@ffff8f518704e680 x1874531146073984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698384.619487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146073984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698384.619492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698384.619494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698384.619495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146073984:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698385.514612:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698386.538704:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698387.562626:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698388.586505:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698389.610640:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698389.738488:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698389.738500:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698389.738505:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698389.738518:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.738531:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698389.738540:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698389.738588:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146074112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698389.738618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698389.738622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698389.738624:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698389.738629:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.738633:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698389.738638:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698389.738650:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698389.738661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146074112:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698389.738667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146074240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698389.738680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698389.738684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698389.738686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698389.738690:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.738694:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698389.738698:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698389.738704:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698389.738717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146074368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698389.738727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698389.738730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698389.738734:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698389.738737:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698389.738742:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.738746:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698389.738751:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698389.738756:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698389.738768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146074496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698389.738778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f5187388380 x1874531146074112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.738778:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698389.738787:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698389.738792:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698389.738799:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00100000:0.0:1787698389.738801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146074112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.738804:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698389.738813:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698389.738815:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698389.738820:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698389.738821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.738826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146074112:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698389.738837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146074624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698389.738847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698389.738850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074240:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698389.738852:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698389.738857:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698389.738862:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.738867:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698389.738871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698389.738875:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698389.738882:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698389.738883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f518738b800 x1874531146074240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698389.738900:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146074752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698389.738912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146074240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.738912:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698389.738920:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698389.738925:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698389.738932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698389.738933:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.738940:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698389.738941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.738949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074240:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698389.738950:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698389.738956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074368:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698389.738957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698389.738974:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146074880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698389.738980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698389.738990:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698389.738992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f518738ad80 x1874531146074368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.738997:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698389.739000:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698389.739005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146074368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.739008:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.739015:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698389.739017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698389.739023:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698389.739030:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698389.739033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698389.739040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698389.739047:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146075008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698389.739071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698389.739084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f518738bb80 x1874531146074496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.739092:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698389.739098:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698389.739100:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698389.739103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146074496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698389.739107:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6289 00000100:00080000:3.0:1787698389.739114:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698389.739121:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698389.739131:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698389.739132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.739144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698389.739146:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146075136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698389.739155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698389.739180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698389.739192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f5187389c00 x1874531146074624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698389.739203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146074624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698389.739211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.739217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146074624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698389.739233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146074752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698389.739252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698389.739263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f518738a300 x1874531146074752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698389.739274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146074752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698389.739282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.739289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146074752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698389.739294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146074880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698389.739311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698389.739321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f5187389500 x1874531146074880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698389.739330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146074880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698389.739337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.739343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146074880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698389.739348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146075008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698389.739390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698389.739400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f5187388a80 x1874531146075008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698389.739410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146075008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698389.739418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.739424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146075008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698389.739429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146075136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698389.739446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698389.739456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698389/real 1787698389] req@ffff8f5187388380 x1874531146075136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698389.739466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146075136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698389.739473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698389.739475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698389.739478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146075136:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698390.634514:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698391.658583:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698392.682613:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698393.706554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698394.730601:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698394.858589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.858597:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698394.858601:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.858609:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.858617:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.858622:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698394.858656:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146075264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698394.858683:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.858685:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698394.858687:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.858690:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.858693:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.858696:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.858704:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698394.858713:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146075392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698394.858720:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.858722:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698394.858725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698394.858727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146075264:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698394.858729:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.858733:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.858737:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.858740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698394.858748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146075520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698394.858760:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.858763:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698394.858765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.858769:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.858772:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.858777:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.858781:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698394.858785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698394.858797:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146075648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698394.858811:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.858814:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698394.858816:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.858822:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00100000:0.0:1787698394.858823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f518738aa00 x1874531146075264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698394.858827:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.858833:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.858844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698394.858849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146075264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698394.858858:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146075776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698394.858863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698394.858871:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698394.858872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698394.858879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787698394.858881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146075264:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698394.858882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.858891:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.858903:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.858907:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.858912:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698394.858925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698394.858926:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146075904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698394.858941:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.858944:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698394.858946:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.858949:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.858954:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698394.858959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698394.858961:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.858982:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698394.858994:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146076032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698394.858997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f5187389180 x1874531146075392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698394.859010:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.859018:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698394.859021:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.859029:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.859033:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698394.859036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146075392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698394.859040:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.859046:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698394.859055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698394.859058:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146076160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698394.859061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075392:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698394.859076:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698394.859081:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698394.859082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075520:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698394.859083:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698394.859092:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6294 00000100:00080000:3.0:1787698394.859097:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698394.859101:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698394.859106:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698394.859113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698394.859118:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146076288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698394.859128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f518738a680 x1874531146075520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146075520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698394.859162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698394.859193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698394.859201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f5187388000 x1874531146075648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146075648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698394.859223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698394.859237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698394.859246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f518738b480 x1874531146075776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146075776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146075776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698394.859287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146075904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698394.859305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698394.859315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f5187389f80 x1874531146075904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146075904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146075904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698394.859343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146076032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698394.859390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698394.859400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f5187388e00 x1874531146076032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146076032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146076032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698394.859430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146076160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698394.859448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698394.859456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f518738b100 x1874531146076160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146076160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146076160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698394.859475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146076288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698394.859485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698394.859491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff8f518738aa00 x1874531146076288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698394.859497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146076288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698394.859502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698394.859503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698394.859504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146076288:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698395.754567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698396.778508:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698397.802561:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698398.826530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698399.850693:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698399.978455:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978465:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698399.978469:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978478:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978487:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978493:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698399.978538:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146076416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698399.978566:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978570:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698399.978572:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978576:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978580:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978584:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698399.978592:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698399.978604:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146076544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698399.978615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698399.978620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978623:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978628:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978633:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698399.978638:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698399.978650:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146076672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698399.978661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698399.978672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698399.978673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146076416:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698399.978681:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978694:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978699:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698399.978708:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698399.978721:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146076800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698399.978730:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698399.978735:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978740:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978743:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978747:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698399.978752:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698399.978761:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146076928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698399.978767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698399.978785:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978789:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698399.978791:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978798:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978801:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978806:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698399.978812:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698399.978822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f5187388380 x1874531146076416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698399.978830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146077056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698399.978848:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698399.978852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146076416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698399.978854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698399.978859:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978872:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00080000:3.0:1787698399.978895:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698399.978900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698399.978909:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698399.978910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.978920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146076416:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698399.978922:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698399.978930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698399.978939:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146077184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698399.978950:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.978954:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698399.978956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698399.978959:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.978968:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00100000:0.0:1787698399.978973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f5187388a80 x1874531146076544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698399.978975:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.978987:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698399.979002:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698399.979006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146076544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698399.979024:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146077312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698399.979027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076544:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698399.979044:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698399.979048:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698399.979050:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698399.979057:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6299 00000100:00100000:0.0:1787698399.979060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076672:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698399.979068:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698399.979076:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698399.979081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698399.979082:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698399.979093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f5187389500 x1874531146076672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698399.979099:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146077440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698399.979120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146076672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698399.979160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698399.979183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698399.979194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f518738a300 x1874531146076800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698399.979203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146076800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698399.979220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698399.979233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698399.979239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f5187389c00 x1874531146076928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698399.979245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146076928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146076928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698399.979257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698399.979267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698399.979272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f518738bb80 x1874531146077056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698399.979279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146077056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698399.979291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698399.979300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698399.979305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f518738ad80 x1874531146077184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698399.979312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146077184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698399.979322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698399.979331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698399.979336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f518738b800 x1874531146077312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698399.979342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146077312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698399.979396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698399.979408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698399.979415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff8f5187388a80 x1874531146077440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698399.979422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146077440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698399.979427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698399.979429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698399.979431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146077440:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698400.874500:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698401.898527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698402.922517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698403.946578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698404.970540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698405.098457:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.098467:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698405.098470:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098480:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098490:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098497:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698405.098529:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146077568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698405.098551:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.098553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698405.098554:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098557:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098559:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098562:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.098570:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698405.098584:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146077696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698405.098589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146077568:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698405.098602:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.098606:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698405.098608:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098613:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098616:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098620:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.098625:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698405.098637:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146077824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698405.098657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698405.098662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698405.098663:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698405.098667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098677:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098681:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.098686:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698405.098698:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146077952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698405.098707:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace90e00 x1874531146077568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698405.098708:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.098717:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698405.098721:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098730:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098734:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098740:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698405.098740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146077568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698405.098748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698405.098758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.098764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.098771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146077568:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698405.098776:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146078080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698405.098787:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698405.098806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146077696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698405.098808:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698405.098813:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098819:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098823:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098827:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.098832:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698405.098838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698405.098847:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146078208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698405.098856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace93100 x1874531146077696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698405.098862:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.098869:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698405.098873:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098878:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00100000:1.0:1787698405.098882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146077696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698405.098884:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698405.098892:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.098900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698405.098901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.098909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.098914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146077696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698405.098918:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146078336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698405.098925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146077824:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698405.098935:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.098939:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698405.098941:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698405.098946:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00080000:0.0:1787698405.098950:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698405.098953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698405.098957:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.098963:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698405.098968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace92680 x1874531146077824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698405.098977:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146078464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698405.098985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146077824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698405.098992:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698405.099000:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698405.099001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698405.099003:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698405.099009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698405.099011:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6304 00000100:00100000:1.0:1787698405.099016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146077824:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698405.099024:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698405.099025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146077952:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698405.099033:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698405.099039:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698405.099051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698405.099054:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146078592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698405.099066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace93b80 x1874531146077952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698405.099079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146077952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698405.099087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.099091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.099094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146077952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698405.099099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146078080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698405.099116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698405.099127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace93800 x1874531146078080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698405.099137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146078080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698405.099144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.099147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.099150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146078080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698405.099165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698405.099184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698405.099195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace90000 x1874531146078208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698405.099207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146078208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698405.099224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.099229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.099233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698405.099239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698405.099259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698405.099269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace90700 x1874531146078336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698405.099281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146078336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698405.099288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.099291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.099294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698405.099299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698405.099316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698405.099326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace91180 x1874531146078464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698405.099336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146078464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698405.099342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.099345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.099349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698405.099382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698405.099399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698405.099409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698405/real 1787698405] req@ffff8f51ace90e00 x1874531146078592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698405.099421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146078592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698405.099428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698405.099431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698405.099434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146078592:192.168.201.125@tcp:8:kworker.0 00010000:00010000:3.0:1787698405.994492:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698407.018539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698408.042489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698409.066517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698410.090558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698410.218484:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698410.218491:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698410.218495:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218502:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218511:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698410.218516:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698410.218544:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146078720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698410.218570:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698410.218574:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698410.218576:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218581:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698410.218588:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698410.218599:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698410.218609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146078720:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698410.218612:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146078848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698410.218629:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698410.218633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698410.218638:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218643:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218647:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698410.218652:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698410.218657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698410.218660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698410.218674:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146078976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698410.218683:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698410.218687:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698410.218688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218695:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698410.218699:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698410.218713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704c700 x1874531146078720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698410.218725:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698410.218733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146078720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698410.218741:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146079104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698410.218742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.218749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.218755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146078720:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698410.218758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698410.218762:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698410.218765:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218770:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218773:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698410.218777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146078848:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698410.218779:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698410.218786:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698410.218799:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146079232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698410.218808:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698410.218813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698410.218814:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698410.218818:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218823:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218826:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698410.218831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704f480 x1874531146078848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698410.218832:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698410.218840:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698410.218847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146078848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698410.218854:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146079360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698410.218855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.218861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.218867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146078848:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698410.218874:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698410.218877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146078976:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698410.218880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698410.218884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.218888:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218892:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698410.218896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698410.218899:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698410.218905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698410.218907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704d500 x1874531146078976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698410.218923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146078976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698410.218923:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146079488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698410.218939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.218944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698410.218945:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698410.218948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146078976:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698410.218951:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698410.218955:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698410.218958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146079104:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698410.218960:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.218966:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698410.218971:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698410.218975:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698410.218988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698410.218989:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146079616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698410.218998:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698410.219002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704d880 x1874531146079104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698410.219003:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698410.219006:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698410.219015:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6309 00000100:00080000:2.0:1787698410.219020:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698410.219026:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698410.219028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146079104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698410.219032:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698410.219036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.219040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.219041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146079104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698410.219047:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146079744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698410.219055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146079232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698410.219071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698410.219080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704e300 x1874531146079232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698410.219089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146079232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698410.219096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.219099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.219102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146079232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698410.219108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698410.219126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698410.219135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704ea00 x1874531146079360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698410.219142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146079360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698410.219151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.219161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.219164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698410.219170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698410.219186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698410.219196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704dc00 x1874531146079488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698410.219207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146079488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698410.219212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.219214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.219215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698410.219219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698410.219231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698410.219236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704ed80 x1874531146079616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698410.219243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146079616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698410.219247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.219249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.219251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698410.219260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698410.219269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698410.219274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698410/real 1787698410] req@ffff8f518704c700 x1874531146079744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698410.219280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146079744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698410.219284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698410.219286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698410.219287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146079744:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698411.114500:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698412.138508:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698413.162518:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787698414.186494:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698415.210533:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698415.338463:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338471:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698415.338475:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338484:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338499:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698415.338532:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146079872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338553:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698415.338555:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338559:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338562:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338566:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338573:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698415.338585:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146080000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698415.338592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146079872:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698415.338593:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338596:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698415.338597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338600:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338602:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338605:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698415.338630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146080128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338637:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338640:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698415.338647:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338650:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338652:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338656:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338660:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698415.338669:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146080256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338680:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698415.338681:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338685:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338687:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338690:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698415.338695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080000:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698415.338698:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698415.338708:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146080384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338716:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338720:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787698415.338720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080128:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698415.338722:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338726:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338728:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338732:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698415.338738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698415.338744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146080512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338752:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338755:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698415.338757:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698415.338759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080384:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698415.338760:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338763:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338767:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698415.338778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698415.338779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146080640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338787:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338790:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698415.338792:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338795:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00100000:1.0:1787698415.338797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080640:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698415.338798:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338802:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338807:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698415.338816:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146080768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698415.338831:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698415.338834:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698415.338836:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698415.338839:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6314 00000100:00080000:0.0:1787698415.338842:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698415.338845:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698415.338849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698415.338849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698415.338858:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146080896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698415.338871:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080896:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698416.234543:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698417.258503:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698418.282492:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698418.410668:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787698418.410708:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698418.410723:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698418.410774:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787698418.410790:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698418.410801:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698418.410812:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698418.410814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace91180 x1874531146080896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698418.410822:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698418.410834:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698418.410849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146080896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698418.410855:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787698418.410871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:2.0:1787698418.410871:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:1.0:1787698418.410881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.410886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698418.410894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace90e00 x1874531146080768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.410908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146080768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.410915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.410919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.410922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698418.410928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace92d80 x1874531146080640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.410939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146080640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.410946:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.410950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.410953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698418.410960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace91880 x1874531146080384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.410971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146080384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.410978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.410981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.410985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698418.410991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace91500 x1874531146080512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.411001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146080512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.411009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.411012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.411030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146080512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698418.411037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace93480 x1874531146080000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.411050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146080000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.411058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.411063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.411066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698418.411072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace90380 x1874531146080128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.411083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146080128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.411091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.411094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.411097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698418.411102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace91c00 x1874531146080256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.411113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146080256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.411121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.411124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.411127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146080256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698418.411132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698415/real 1787698418] req@ffff8f51ace92300 x1874531146079872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698418.411143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146079872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698418.411150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698418.411153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698418.411156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146079872:192.168.201.125@tcp:250:kworker.0 00010000:00010000:1.0:1787698419.306480:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698420.330540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698420.458508:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.458517:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698420.458521:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.458540:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.458550:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.458557:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698420.458589:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146081024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698420.458657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146081024:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698420.458692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.458696:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698420.458698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.458702:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.458706:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.458709:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698420.458717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698420.458722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698420.458731:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146081152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698420.458740:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.458743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698420.458744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.458748:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.458751:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.458756:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698420.458760:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698420.458780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146081280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698420.458788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f5187388380 x1874531146081024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698420.458793:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.458801:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698420.458804:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.458810:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.458814:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.458818:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698420.458824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146081024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698420.458827:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698420.458842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698420.458845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146081408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698420.458850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.458859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146081024:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698420.458865:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.458870:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698420.458872:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.458878:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.458882:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.458888:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698420.458893:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698420.458901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698420.458909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146081536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698420.458920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.458924:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698420.458926:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698420.458931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698420.458933:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.458939:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.458946:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698420.458951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f518738aa00 x1874531146081152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698420.458951:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698420.458970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146081152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698420.458976:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146081664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698420.458982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.458990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698420.458995:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698420.458997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081152:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698420.459000:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698420.459005:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698420.459007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698420.459012:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.459018:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698420.459022:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698420.459027:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698420.459034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698420.459042:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146081792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698420.459051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f518738b100 x1874531146081280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698420.459054:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.459062:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698420.459066:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.459072:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.459076:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698420.459078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146081280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698420.459082:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698420.459090:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698420.459092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698420.459105:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146081920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698420.459112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081408:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698420.459123:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698420.459126:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698420.459128:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698420.459133:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6320 00000100:00080000:3.0:1787698420.459137:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698420.459141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698420.459143:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698420.459150:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698420.459156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f5187388e00 x1874531146081408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698420.459166:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146082048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698420.459175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146081408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698420.459186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698420.459213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698420.459231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698420.459242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f5187389f80 x1874531146081536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698420.459259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146081536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698420.459267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146081536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698420.459279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146081664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698420.459296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698420.459306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f518738b480 x1874531146081664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698420.459316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146081664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698420.459324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146081664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698420.459336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146081792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698420.459385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698420.459397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f5187388000 x1874531146081792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698420.459407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146081792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698420.459419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146081792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698420.459431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146081920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698420.459450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698420.459459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f518738a680 x1874531146081920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698420.459469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146081920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698420.459475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146081920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698420.459486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146082048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698420.459501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698420.459511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698420/real 1787698420] req@ffff8f5187389180 x1874531146082048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698420.459520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146082048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698420.459538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698420.459542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698420.459545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146082048:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698421.354486:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698422.378524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698423.402531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698424.426615:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698425.450512:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698425.578524:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578538:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698425.578543:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578553:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578566:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578573:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698425.578615:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146082176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578639:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578642:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698425.578644:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578647:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578649:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578652:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698425.578670:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146082304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578680:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698425.578681:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578683:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578684:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698425.578689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146082176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698425.578690:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578697:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698425.578704:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146082432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578710:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698425.578714:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578716:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578718:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578720:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698425.578732:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146082560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578738:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578740:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698425.578741:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578743:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578753:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578755:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578757:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698425.578762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698425.578766:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146082688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578773:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578775:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698425.578776:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578780:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578781:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578784:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578787:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698425.578793:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146082816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578799:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578801:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698425.578803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578806:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578808:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578811:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578814:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698425.578825:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146082944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698425.578828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace90700 x1874531146082176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698425.578845:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578850:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698425.578852:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578857:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00080000:0.0:1787698425.578861:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698425.578864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146082176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698425.578869:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578878:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698425.578883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.578887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.578891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146082176:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698425.578894:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146083072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698425.578905:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698425.578909:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698425.578911:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698425.578915:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6325 00000100:00100000:1.0:1787698425.578917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082304:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698425.578923:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698425.578929:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698425.578934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698425.578942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698425.578947:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146083200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698425.578954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace90000 x1874531146082304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.578966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146082304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.578971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.578974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.578977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698425.578984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698425.579004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace93800 x1874531146082432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146082432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698425.579048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698425.579068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace93b80 x1874531146082560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146082560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698425.579108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698425.579123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace92680 x1874531146082688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146082688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146082688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698425.579171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146082816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698425.579190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace93100 x1874531146082816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146082816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146082816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698425.579234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146082944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698425.579254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace92300 x1874531146082944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146082944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146082944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698425.579302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146083072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698425.579321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace91c00 x1874531146083072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146083072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146083072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698425.579408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146083200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698425.579429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698425.579451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698425/real 1787698425] req@ffff8f51ace90380 x1874531146083200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698425.579464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146083200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698425.579471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698425.579472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698425.579475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146083200:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698426.474616:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698427.498579:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698428.522621:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698429.546583:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698430.570563:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698430.698518:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698527:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698430.698531:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698539:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698549:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698555:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698430.698586:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146083328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698430.698615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698618:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698430.698620:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698623:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698625:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698635:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698430.698644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146083456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698430.698653:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698655:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698430.698657:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698661:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698663:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698666:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698430.698676:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146083584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698430.698685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698430.698690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698430.698691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146083328:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698430.698695:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698703:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698708:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698711:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698430.698723:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146083712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698430.698729:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698731:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698430.698732:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698742:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698745:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698751:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698430.698763:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146083840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698430.698771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698773:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698430.698775:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698777:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698779:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698781:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698785:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698430.698800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146083968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698430.698802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698430.698815:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698430.698820:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698823:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698827:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.698832:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698840:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698430.698853:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146084096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698430.698859:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698430.698861:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698430.698863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698430.698865:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.698870:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698430.698875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183644700 x1874531146083328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698430.698876:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.698889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698430.698914:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146084224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698430.698919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146083328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.698977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698430.698980:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698430.698983:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.698993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146083328:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698430.698995:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698430.699002:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698430.699008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083456:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698430.699011:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6330 00000100:00080000:1.0:1787698430.699020:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698430.699027:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698430.699033:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698430.699039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698430.699052:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146084352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698430.699056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183644000 x1874531146083456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146083456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698430.699155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698430.699167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183646680 x1874531146083584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146083584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698430.699192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698430.699202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183647800 x1874531146083712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146083712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698430.699225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698430.699233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183647480 x1874531146083840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146083840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146083840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698430.699257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146083968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698430.699265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183645c00 x1874531146083968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146083968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146083968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698430.699288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146084096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698430.699298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183645180 x1874531146084096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146084096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146084096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698430.699326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146084224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698430.699335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183647100 x1874531146084224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146084224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146084224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698430.699397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146084352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698430.699409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698430.699416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698430/real 1787698430] req@ffff8f5183644380 x1874531146084352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698430.699422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146084352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698430.699427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698430.699429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698430.699432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146084352:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698431.594535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698432.618541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698433.642529:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698434.666545:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698435.690482:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698435.818494:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818502:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698435.818505:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818514:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818523:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818529:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698435.818559:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146084480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698435.818581:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818583:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698435.818591:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818595:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818597:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818609:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698435.818618:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146084608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698435.818623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146084480:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698435.818631:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818635:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698435.818636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818639:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818641:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818644:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818648:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698435.818657:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146084736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698435.818663:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698435.818667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818671:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818673:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818676:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698435.818683:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146084864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698435.818689:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698435.818694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698435.818696:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818700:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818704:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818707:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818710:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698435.818718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146084992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698435.818724:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818727:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698435.818729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698435.818733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f5187389180 x1874531146084480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698435.818735:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818741:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818748:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698435.818759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146084480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.818772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.818778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698435.818778:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146085120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698435.818785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146084480:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698435.818795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698435.818800:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818803:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818806:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818809:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818812:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698435.818813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698435.818824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146085248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698435.818832:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818836:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698435.818839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698435.818839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698435.818846:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818849:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818852:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698435.818857:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698435.818862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f518738a680 x1874531146084608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698435.818866:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146085376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698435.818894:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698435.818897:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698435.818898:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698435.818901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146084608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698435.818904:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6335 00000100:00080000:3.0:1787698435.818910:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698435.818915:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698435.818918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698435.818920:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698435.818924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698435.818930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146085504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698435.818932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698435.818950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698435.818976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.818997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f5187388000 x1874531146084736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146084736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698435.819021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698435.819037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.819047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f518738b480 x1874531146084864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146084864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698435.819082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698435.819101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.819125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f5187389f80 x1874531146084992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146084992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146084992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698435.819162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698435.819177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.819185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f5187388e00 x1874531146085120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146085120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698435.819224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698435.819236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.819242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f518738b100 x1874531146085248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146085248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698435.819288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698435.819308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.819319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f518738aa00 x1874531146085376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146085376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698435.819352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698435.819404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698435.819414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698435/real 1787698435] req@ffff8f5187388380 x1874531146085504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698435.819422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146085504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698435.819428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698435.819429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698435.819432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146085504:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698436.714518:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698437.738510:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698438.762525:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698439.786570:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698440.810541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698440.938397:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938406:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698440.938410:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938418:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938427:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938433:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698440.938467:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146085632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698440.938492:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938495:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698440.938497:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938500:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938503:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938505:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938512:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698440.938521:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146085760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698440.938527:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938529:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698440.938531:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938533:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938535:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938537:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938541:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698440.938548:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146085888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698440.938555:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938559:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698440.938561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146085632:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698440.938562:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938569:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938572:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938574:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938578:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698440.938588:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146086016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698440.938599:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938601:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698440.938603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938605:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938608:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938609:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698440.938620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146086144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698440.938627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698440.938631:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938635:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698440.938636:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938640:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938643:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938646:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938649:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698440.938656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146086272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698440.938662:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698440.938666:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938669:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938672:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938675:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938678:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698440.938681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f5187388a80 x1874531146085632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698440.938688:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146086400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698440.938706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.938708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698440.938710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938715:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00100000:0.0:1787698440.938716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146085632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698440.938719:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938725:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938733:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698440.938736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.938741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698440.938743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146086528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698440.938747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146085632:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698440.938759:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698440.938760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146085760:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698440.938765:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698440.938769:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.938773:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6340 00000100:00080000:3.0:1787698440.938775:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698440.938777:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698440.938781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698440.938793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698440.938794:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146086656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698440.938807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f518738b800 x1874531146085760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.938818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146085760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.938825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.938828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.938831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146085760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698440.938851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146085888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698440.938864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.938870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f518738ad80 x1874531146085888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.938878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146085888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.938883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.938885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.938887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146085888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698440.938890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146086016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698440.938901:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.938906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f518738bb80 x1874531146086016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.938914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146086016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.938919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.938920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.938922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146086016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698440.938926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146086144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698440.938935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.938940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f5187389c00 x1874531146086144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.938948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146086144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.938953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.938955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.938957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146086144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698440.938960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698440.938970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.938985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f518738a300 x1874531146086272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.938993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146086272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.938998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.939000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.939002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698440.939006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698440.939017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.939023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f5187389500 x1874531146086400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.939030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146086400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.939039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.939041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.939043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698440.939046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698440.939055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.939061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f5187388380 x1874531146086528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.939067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146086528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.939072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.939073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.939075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698440.939078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698440.939087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698440.939093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff8f518738aa00 x1874531146086656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698440.939100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146086656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698440.939104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698440.939106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698440.939109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146086656:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698441.834540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698442.858494:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698443.882519:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698444.906489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698445.930524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698446.058416:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058424:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698446.058428:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698446.058436:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058446:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698446.058452:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698446.058492:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146086784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698446.058515:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058517:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698446.058519:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698446.058523:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058525:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698446.058528:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058535:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698446.058544:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146086912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698446.058552:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058555:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698446.058556:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698446.058561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146086784:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698446.058562:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058571:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698446.058576:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058581:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698446.058591:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146087040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698446.058600:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058603:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698446.058605:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698446.058609:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058613:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698446.058617:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698446.058627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698446.058639:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146087168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698446.058651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058655:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698446.058657:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698446.058662:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058666:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698446.058671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f518738b100 x1874531146086784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698446.058672:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058681:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698446.058692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146086784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698446.058695:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146087296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698446.058719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698446.058720:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698446.058728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698446.058729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698446.058735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146086784:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698446.058736:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058742:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698446.058746:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698446.058759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146087424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698446.058760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146086912:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698446.058776:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058779:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698446.058782:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698446.058787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698446.058791:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058797:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698446.058801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f5187388e00 x1874531146086912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698446.058805:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058816:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698446.058819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146086912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.058827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698446.058829:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146087552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698446.058833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.058842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146086912:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698446.058844:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058851:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698446.058855:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698446.058864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146087040:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698446.058864:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058870:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698446.058877:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058882:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698446.058894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698446.058899:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146087680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698446.058911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f5187389f80 x1874531146087040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698446.058912:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698446.058920:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698446.058923:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698446.058932:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6345 00000100:00080000:3.0:1787698446.058937:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698446.058942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146087040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698446.058945:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698446.058952:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698446.058962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698446.058966:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146087808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698446.058970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.058980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146087040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698446.058990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146087168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698446.059011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698446.059019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f518738b480 x1874531146087168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698446.059027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146087168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.059042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698446.059044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.059047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146087168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698446.059051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146087296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698446.059063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698446.059070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f5187388000 x1874531146087296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698446.059078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146087296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.059086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698446.059118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.059123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146087296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698446.059143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698446.059160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698446.059168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f518738a680 x1874531146087424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698446.059176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146087424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.059182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698446.059184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.059187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698446.059190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698446.059200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698446.059206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f5187389180 x1874531146087552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698446.059220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146087552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.059230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698446.059232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.059234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698446.059237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698446.059248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698446.059254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f518738b100 x1874531146087680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698446.059261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146087680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.059271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698446.059273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.059275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698446.059279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698446.059289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698446.059295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698446/real 1787698446] req@ffff8f518738aa00 x1874531146087808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698446.059303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146087808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698446.059307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698446.059309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698446.059312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146087808:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698446.954556:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698447.978550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698449.002511:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698450.026540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698451.050526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698451.178517:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.178525:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698451.178529:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.178537:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178546:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698451.178552:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698451.178583:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146087936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698451.178621:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.178625:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698451.178627:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.178632:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178637:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698451.178642:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.178652:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698451.178668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146087936:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698451.178669:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146088064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698451.178690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.178694:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698451.178696:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.178700:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178704:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698451.178709:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.178714:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698451.178726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146088192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698451.178734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698451.178740:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.178744:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698451.178747:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.178752:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178756:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698451.178760:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.178766:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698451.178772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace93480 x1874531146087936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698451.178780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146088320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698451.178792:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698451.178792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146087936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698451.178802:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698451.178807:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698451.178809:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698451.178813:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178817:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698451.178823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698451.178826:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698451.178831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146087936:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698451.178833:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698451.178845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146088448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698451.178853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088064:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698451.178866:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.178869:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698451.178871:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.178875:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178879:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698451.178879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698451.178885:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.178889:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698451.178894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace91500 x1874531146088064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698451.178900:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146088576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698451.178911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146088064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698451.178912:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698451.178921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.178927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698451.178929:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698451.178934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088064:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698451.178937:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698451.178943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088192:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698451.178945:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178953:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698451.178959:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.178965:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698451.178975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698451.178980:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146088704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698451.178988:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.178990:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698451.178992:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.178995:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:0.0:1787698451.178999:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698451.179000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace91880 x1874531146088192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698451.179019:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.179026:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698451.179037:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146088832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698451.179045:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698451.179048:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698451.179048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146088192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698451.179052:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698451.179060:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6350 00000100:00080000:1.0:1787698451.179067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698451.179071:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698451.179075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698451.179078:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698451.179083:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698451.179084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698451.179094:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146088960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698451.179096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698451.179128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698451.179138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace92d80 x1874531146088320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698451.179147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146088320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698451.179152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.179154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698451.179156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698451.179167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698451.179179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698451.179185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace90e00 x1874531146088448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698451.179192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146088448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698451.179202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.179204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698451.179207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146088448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698451.179210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698451.179220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698451.179225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace91180 x1874531146088576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698451.179232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146088576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698451.179237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.179239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698451.179241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698451.179245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698451.179254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698451.179259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace93480 x1874531146088704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698451.179266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146088704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698451.179271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.179273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698451.179275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698451.179279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698451.179288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698451.179293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace90380 x1874531146088832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698451.179300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146088832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698451.179305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.179307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698451.179309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698451.179312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698451.179320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698451.179326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698451/real 1787698451] req@ffff8f51ace91c00 x1874531146088960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698451.179333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146088960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698451.179338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698451.179340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698451.179341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146088960:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698452.074566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698453.098514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698454.122559:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698455.146480:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698456.170572:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698456.298462:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298471:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698456.298475:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298483:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00080000:3.0:1787698456.298493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298499:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698456.298532:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146089088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698456.298554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298557:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698456.298558:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298561:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00080000:3.0:1787698456.298563:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298568:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698456.298578:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698456.298593:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146089216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698456.298598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146089088:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698456.298615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298619:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698456.298621:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298626:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00080000:3.0:1787698456.298630:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298634:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698456.298638:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698456.298649:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146089344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698456.298657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298659:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698456.298660:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298665:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6355 00000800:00000100:1.0:1787698456.298668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698456.298669:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298674:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698456.298677:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698456.298686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146089472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698456.298692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298695:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698456.298696:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298699:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00080000:3.0:1787698456.298702:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698456.298706:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698456.298714:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146089600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698456.298720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f5187388380 x1874531146089088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698456.298722:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698456.298735:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298743:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00080000:3.0:1787698456.298745:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298748:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698456.298751:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698456.298754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146089088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698456.298762:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146089728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698456.298772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698456.298774:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298779:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698456.298781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698456.298782:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698456.298789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146089088:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698456.298790:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00080000:3.0:1787698456.298797:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298803:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698456.298810:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698456.298822:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146089856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698456.298826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698456.298843:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698456.298847:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698456.298850:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698456.298855:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6355 00000800:00000100:1.0:1787698456.298859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698456.298862:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298870:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698456.298876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f5187389500 x1874531146089216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698456.298876:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698456.298895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146089984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698456.298896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146089216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698456.298931:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698456.298938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698456.298938:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698456.298942:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698456.298947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698456.298951:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6355 00000100:00100000:1.0:1787698456.298958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089216:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698456.298958:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698456.298967:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698456.298973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089344:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698456.298974:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698456.298994:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146090112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698456.299005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f518738a300 x1874531146089344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146089344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698456.299051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698456.299066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f5187389c00 x1874531146089472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146089472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698456.299094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698456.299104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f518738bb80 x1874531146089600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146089600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146089600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698456.299138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146089728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698456.299153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f518738ad80 x1874531146089728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146089728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146089728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698456.299199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146089856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698456.299220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f518738b800 x1874531146089856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146089856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146089856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698456.299269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146089984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698456.299291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f5187388a80 x1874531146089984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146089984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146089984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698456.299329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146090112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698456.299343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698456.299350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698456/real 1787698456] req@ffff8f5187388380 x1874531146090112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698456.299385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146090112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698456.299394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698456.299397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698456.299400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146090112:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698457.194536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698458.218510:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698459.242542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698460.266532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698461.290565:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698461.418439:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418447:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698461.418452:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418461:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418472:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418480:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698461.418519:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146090240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698461.418547:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418550:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698461.418551:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418555:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418561:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418569:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698461.418580:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146090368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698461.418588:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418590:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698461.418591:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418594:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418596:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418598:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418603:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698461.418606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146090240:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698461.418619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146090496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698461.418637:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698461.418642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418646:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418650:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698461.418672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146090624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698461.418681:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698461.418688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698461.418689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698461.418693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418698:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418702:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418705:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418710:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698461.418718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146090752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698461.418725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418728:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698461.418730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418736:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00100000:1.0:1787698461.418739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f518738aa00 x1874531146090240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698461.418742:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418752:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418759:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698461.418764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146090240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.418775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698461.418776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146090880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698461.418780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.418788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146090240:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698461.418797:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698461.418799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090368:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698461.418802:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698461.418806:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418810:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418814:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418817:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698461.418845:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146091008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698461.418853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698461.418863:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418866:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698461.418869:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698461.418874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f518738b100 x1874531146090368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698461.418876:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00080000:3.0:1787698461.418884:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418892:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698461.418910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146090368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.418921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698461.418923:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146091136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698461.418930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.418939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090368:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698461.418956:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698461.418960:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698461.418962:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698461.418968:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6361 00000100:00100000:1.0:1787698461.418970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090496:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698461.418975:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698461.418983:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698461.418989:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698461.418994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f5187389180 x1874531146090496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698461.419011:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146091264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698461.419047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146090496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698461.419061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698461.419080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f518738a680 x1874531146090624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698461.419096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146090624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698461.419109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698461.419121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f5187388000 x1874531146090752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698461.419143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146090752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146090752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698461.419170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146090880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698461.419190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f518738b480 x1874531146090880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698461.419222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146090880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146090880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698461.419245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146091008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698461.419266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f5187389f80 x1874531146091008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698461.419283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146091008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146091008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698461.419300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146091136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698461.419319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f5187388e00 x1874531146091136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698461.419343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146091136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146091136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698461.419402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146091264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698461.419419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698461.419427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698461/real 1787698461] req@ffff8f518738b100 x1874531146091264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698461.419446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146091264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698461.419452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698461.419454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698461.419457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146091264:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698462.314535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698463.338533:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698464.362505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698465.386539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698466.410563:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698466.538541:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.538553:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698466.538557:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.538567:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.538579:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.538586:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698466.538621:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146091392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698466.538656:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.538660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698466.538662:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.538668:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.538672:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.538677:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698466.538689:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698466.538705:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146091520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698466.538721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146091392:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698466.538721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.538731:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698466.538735:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.538751:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.538756:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.538760:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698466.538767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698466.538782:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146091648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698466.538793:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.538797:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698466.538800:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.538804:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.538818:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.538821:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698466.538826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698466.538833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698466.538842:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146091776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698466.538852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.538855:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698466.538858:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.538861:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.538863:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.538866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698466.538870:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698466.538885:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146091904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698466.538893:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183647b80 x1874531146091392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698466.538895:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.538909:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698466.538916:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.538929:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.538934:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.538938:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698466.538943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146091392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698466.538946:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698466.538968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698466.538968:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146092032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698466.538981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.538993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146091392:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698466.538995:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.539009:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698466.539011:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.539016:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00080000:1.0:1787698466.539018:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.539021:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698466.539024:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698466.539035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698466.539040:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146092160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698466.539052:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.539054:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698466.539058:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698466.539065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698466.539071:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6366 00000100:00100000:0.0:1787698466.539078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183646a00 x1874531146091520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698466.539089:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698466.539093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146091520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698466.539100:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698466.539106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787698466.539114:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698466.539119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698466.539136:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146092288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698466.539144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091648:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698466.539160:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698466.539165:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698466.539168:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698466.539174:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6366 00000800:00000100:0.0:1787698466.539177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698466.539185:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698466.539193:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183646d80 x1874531146091648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698466.539203:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698466.539225:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146092416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698466.539228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146091648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698466.539321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698466.539339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183644e00 x1874531146091776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698466.539389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146091776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698466.539403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698466.539419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183644380 x1874531146091904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698466.539442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146091904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146091904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698466.539461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698466.539476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183647100 x1874531146092032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698466.539490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146092032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698466.539514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698466.539524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183645180 x1874531146092160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698466.539536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146092160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698466.539548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698466.539557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183645c00 x1874531146092288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698466.539570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146092288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698466.539580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698466.539589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698466.539595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698466/real 1787698466] req@ffff8f5183647480 x1874531146092416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698466.539601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146092416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698466.539606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698466.539607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698466.539609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146092416:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698467.434552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698468.458545:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698469.482538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698470.506526:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698471.530554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698471.658490:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698471.658499:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698471.658502:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698471.658511:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.658521:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.658529:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698471.658568:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146092544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698471.658622:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698471.658625:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698471.658627:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698471.658631:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.658634:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.658637:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.658711:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698471.658719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146092544:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787698471.658784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698471.658820:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146092672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698471.658832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f518738aa00 x1874531146092544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698471.658866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146092544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698471.658875:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698471.658882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.658889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.658894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146092544:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698471.658896:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698471.658906:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698471.658920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146092672:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698471.658928:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.658934:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.658940:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.658946:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698471.658952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698471.658966:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146092800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698471.658969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f5187388380 x1874531146092672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698471.658999:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698471.659004:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698471.659005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146092672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698471.659006:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698471.659015:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.659021:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698471.659025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698471.659026:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698471.659036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698471.659037:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698471.659045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146092672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698471.659051:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146092928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698471.659060:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698471.659063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146092800:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698471.659064:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698471.659067:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698471.659070:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.659072:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.659085:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.659089:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698471.659101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698471.659102:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146093056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698471.659112:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698471.659115:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698471.659116:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698471.659120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f5187388a80 x1874531146092800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698471.659122:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.659127:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.659134:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.659141:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698471.659146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146092800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698471.659152:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146093184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698471.659179:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698471.659184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698471.659184:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698471.659188:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698471.659191:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.659193:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.659195:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.659199:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698471.659201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146092800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698471.659209:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146093312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698471.659218:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698471.659221:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698471.659222:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698471.659225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146092928:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698471.659229:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00080000:3.0:1787698471.659234:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.659237:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.659240:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698471.659249:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146093440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698471.659257:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698471.659260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698471.659265:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698471.659269:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698471.659274:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6371 00000100:00100000:0.0:1787698471.659280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f518738b800 x1874531146092928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698471.659281:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698471.659291:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698471.659301:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698471.659307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146092928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698471.659317:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146093568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698471.659319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.659328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146092928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698471.659347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146093056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698471.659406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698471.659419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f518738ad80 x1874531146093056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698471.659432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146093056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698471.659441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.659444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146093056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698471.659454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698471.659471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698471.659479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f518738bb80 x1874531146093184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698471.659487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146093184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698471.659493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.659495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698471.659500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698471.659510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698471.659516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f5187389c00 x1874531146093312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698471.659523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146093312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698471.659527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.659529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698471.659543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698471.659552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698471.659559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f518738a300 x1874531146093440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698471.659566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146093440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698471.659571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.659573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698471.659578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698471.659588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698471.659594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698471/real 1787698471] req@ffff8f5187389500 x1874531146093568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698471.659601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146093568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698471.659606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698471.659607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698471.659609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146093568:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698472.554494:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698473.578582:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698474.602550:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698475.626579:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698476.650559:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698476.778534:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778547:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698476.778551:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778594:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778608:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778618:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698476.778664:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146093696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778697:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778700:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698476.778702:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778705:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778708:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778711:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.778719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698476.778728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146093824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698476.778738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778741:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778743:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778745:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.778747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698476.778753:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146093952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778762:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698476.778762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146093696:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698476.778767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698476.778771:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778776:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778778:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778779:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.778782:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698476.778790:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146094080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778797:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698476.778798:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778801:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778803:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778805:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.778807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698476.778814:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146094208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778819:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778821:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698476.778822:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778824:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778834:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778836:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.778839:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698476.778847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698476.778849:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146094336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778858:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778861:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698476.778863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778868:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778873:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.778878:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.778883:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698476.778898:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146094464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698476.778903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace92300 x1874531146093696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698476.778913:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778924:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698476.778928:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.778933:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6376 00000100:00080000:0.0:1787698476.778938:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698476.778939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146093696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698476.778946:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698476.778955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.778962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787698476.778965:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698476.778968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146093696:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698476.778981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146093824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698476.778985:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146094592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698476.778994:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698476.778996:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698476.778998:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698476.779002:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6376 00000800:00000100:1.0:1787698476.779007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698476.779008:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698476.779016:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698476.779021:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698476.779024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace93100 x1874531146093824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698476.779037:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146094720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698476.779041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146093824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146093824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698476.779099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146093952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698476.779118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace92680 x1874531146093952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146093952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146093952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698476.779159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146094080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698476.779177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace93b80 x1874531146094080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146094080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146094080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698476.779215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146094208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698476.779232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace93800 x1874531146094208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146094208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146094208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698476.779274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698476.779291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace90000 x1874531146094336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146094336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698476.779333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698476.779348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace90700 x1874531146094464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146094464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698476.779424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698476.779442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace91c00 x1874531146094592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146094592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698476.779483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698476.779498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698476.779507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698476/real 1787698476] req@ffff8f51ace90380 x1874531146094720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698476.779519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146094720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698476.779527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698476.779530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698476.779534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146094720:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698477.674591:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698478.698564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698479.722506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698480.746537:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698481.770492:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698481.898512:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898522:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698481.898526:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.898536:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898547:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898554:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698481.898588:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146094848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.898613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698481.898618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.898623:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898626:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898630:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898638:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698481.898650:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146094976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.898660:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698481.898666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146094848:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698481.898666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698481.898668:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.898676:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898680:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898683:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898687:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698481.898697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146095104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.898711:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898722:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698481.898723:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.898727:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898730:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898734:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898738:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698481.898749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146095232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.898758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898761:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698481.898763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698481.898768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146094976:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698481.898769:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898775:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898780:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898792:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698481.898806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146095104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698481.898807:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146095360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.898822:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898826:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698481.898828:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.898833:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898837:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698481.898838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146095232:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698481.898846:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698481.898866:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146095488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698481.898870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146095360:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698481.898884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898888:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698481.898898:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698481.898900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095488:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698481.898906:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.898910:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898913:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898917:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698481.898927:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146095616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.898936:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.898939:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698481.898941:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.898946:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00100000:1.0:1787698481.898950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095616:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698481.898951:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.898960:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.898965:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698481.898977:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146095744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698481.899006:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698481.899010:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698481.899011:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698481.899015:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6381 00000100:00080000:3.0:1787698481.899019:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698481.899023:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698481.899028:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698481.899037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698481.899042:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146095872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698481.899074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095872:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698482.794539:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698483.818498:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698484.842549:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698484.970627:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787698484.970667:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698484.970681:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970724:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970739:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970747:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970759:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970767:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698484.970773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f518738aa00 x1874531146095872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698484.970775:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970786:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698484.970797:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698484.970813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146095872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.970832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.970837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.970842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.970850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f5187388380 x1874531146095744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.970862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146095744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.970875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.970880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.970883:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.970890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f5187388a80 x1874531146095616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.970902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146095616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.970917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.970920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.970924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698484.970931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f518738b800 x1874531146095488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.970942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146095488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.970965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.970968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.970972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146095488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698484.970979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f518738ad80 x1874531146095360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.970991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146095360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.970998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.971001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.971004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146095360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.971011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f518738a300 x1874531146094976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.971022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146094976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.971030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.971033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.971036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146094976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698484.971042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f5187389c00 x1874531146095104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.971052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146095104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.971059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.971062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.971066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146095104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698484.971072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f518738bb80 x1874531146095232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.971082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146095232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.971090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.971092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.971095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146095232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698484.971101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698484] req@ffff8f5187389500 x1874531146094848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698484.971111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146094848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698484.971119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698484.971122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698484.971125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146094848:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787698485.866528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698486.890565:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698487.018451:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698487.018462:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018479:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698487.018484:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698487.018513:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146096000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698487.018535:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018537:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698487.018538:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018541:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018543:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698487.018545:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018552:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698487.018560:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146096128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698487.018568:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698487.018569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146096000:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698487.018571:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698487.018575:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018578:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018580:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698487.018582:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698487.018592:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146096256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698487.018597:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018600:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698487.018601:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018604:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018617:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698487.018621:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018626:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698487.018627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698487.018640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146096384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698487.018649:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698487.018654:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018657:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018661:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698487.018664:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018669:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698487.018673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183647800 x1874531146096000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698487.018684:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146096512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698487.018692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146096000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698487.018696:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698487.018708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698487.018711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698487.018714:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:0.0:1787698487.018718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698487.018720:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698487.018725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146096000:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698487.018728:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698487.018750:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146096640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698487.018758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096128:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698487.018762:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698487.018769:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018774:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018779:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698487.018783:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698487.018791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698487.018804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146096768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698487.018810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183646680 x1874531146096128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698487.018817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018825:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698487.018828:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698487.018830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146096128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698487.018835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018842:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698487.018846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698487.018848:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698487.018853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787698487.018856:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698487.018859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698487.018871:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146096896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698487.018873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096256:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698487.018888:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698487.018891:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698487.018893:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698487.018896:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6386 00000100:00080000:1.0:1787698487.018901:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698487.018903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698487.018908:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698487.018914:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698487.018920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183644000 x1874531146096256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698487.018927:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146097024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698487.018941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146096256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.018951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.018954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.018958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698487.018965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698487.018989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698487.019000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183644700 x1874531146096384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698487.019010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146096384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.019023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.019027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.019030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698487.019037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698487.019057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698487.019067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183647480 x1874531146096512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698487.019076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146096512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.019097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.019100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.019103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146096512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698487.019119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146096640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698487.019137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698487.019147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183645c00 x1874531146096640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698487.019157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146096640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.019164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.019167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.019171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146096640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698487.019176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146096768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698487.019201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698487.019211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183645180 x1874531146096768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698487.019222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146096768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.019259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.019263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.019266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146096768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698487.019272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146096896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698487.019291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698487.019301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183647100 x1874531146096896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698487.019310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146096896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.019316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.019319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.019321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146096896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698487.019327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146097024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698487.019343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698487.019352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698487/real 1787698487] req@ffff8f5183644380 x1874531146097024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698487.019435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146097024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698487.019443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698487.019446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698487.019448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146097024:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698487.914527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698488.938536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698489.962525:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698490.986549:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698492.010518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698492.138477:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138484:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698492.138487:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698492.138494:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138502:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138507:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698492.138535:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146097152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698492.138558:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138561:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698492.138562:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698492.138565:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138571:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138578:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698492.138589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146097152:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698492.138597:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146097280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698492.138604:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138606:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698492.138608:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698492.138611:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138614:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138617:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698492.138632:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146097408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698492.138641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787698492.138647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698492.138648:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698492.138655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138660:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138663:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138667:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698492.138677:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146097536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698492.138683:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138686:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698492.138686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace93480 x1874531146097152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698492.138689:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698492.138696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138701:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138703:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138706:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698492.138708:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146097152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698492.138715:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146097664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698492.138721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698492.138724:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698492.138727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698492.138728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698492.138731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698492.138734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146097152:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698492.138737:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138743:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138746:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698492.138754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146097792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698492.138758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698492.138765:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138768:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698492.138770:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698492.138773:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00080000:0.0:1787698492.138777:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698492.138780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698492.138784:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698492.138795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace91180 x1874531146097280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698492.138798:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146097920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698492.138817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146097280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698492.138819:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698492.138829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698492.138830:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698492.138834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698492.138841:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00100000:1.0:1787698492.138841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698492.138850:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698492.138851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097408:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698492.138859:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138864:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698492.138868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698492.138876:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146098048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698492.138879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace90e00 x1874531146097408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.138895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146097408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698492.138901:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698492.138905:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698492.138905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698492.138909:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698492.138911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698492.138915:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6391 00000100:00100000:1.0:1787698492.138919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097408:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698492.138920:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698492.138926:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698492.138931:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698492.138935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698492.138941:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146098176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698492.138963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698492.138983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace92d80 x1874531146097536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.138990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146097536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698492.138995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698492.138997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698492.138998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698492.139002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698492.139013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698492.139019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace91880 x1874531146097664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.139025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146097664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698492.139029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698492.139031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698492.139033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146097664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698492.139042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146097792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698492.139051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698492.139062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace91500 x1874531146097792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.139073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146097792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698492.139078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698492.139080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698492.139082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146097792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698492.139084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146097920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698492.139094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698492.139099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace93480 x1874531146097920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.139106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146097920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698492.139110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698492.139112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698492.139113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146097920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698492.139117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146098048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698492.139126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698492.139130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace90380 x1874531146098048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.139137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146098048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698492.139145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698492.139147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698492.139149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146098048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698492.139151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146098176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698492.139159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698492.139163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698492/real 1787698492] req@ffff8f51ace91c00 x1874531146098176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698492.139169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146098176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698492.139173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698492.139175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698492.139177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146098176:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698493.034539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698494.058481:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698495.082506:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698496.106466:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698497.130499:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698497.258420:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258427:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698497.258430:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258437:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258445:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258451:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698497.258487:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146098304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698497.258505:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258507:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698497.258508:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258511:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258513:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258515:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698497.258523:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698497.258530:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146098432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698497.258536:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258539:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698497.258540:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258542:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258544:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258546:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698497.258551:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698497.258555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146098304:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698497.258564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146098560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698497.258581:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258585:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698497.258586:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258591:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258605:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258608:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698497.258612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698497.258616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698497.258624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146098688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698497.258632:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258634:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698497.258635:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258638:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258640:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258642:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698497.258645:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698497.258652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146098816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698497.258659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace90700 x1874531146098304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698497.258676:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258679:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698497.258680:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258684:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00100000:1.0:1787698497.258688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146098304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698497.258689:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258695:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698497.258702:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698497.258711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698497.258714:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146098944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698497.258717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.258723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146098304:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698497.258727:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698497.258730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698497.258733:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258738:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258740:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698497.258743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098432:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698497.258745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698497.258755:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146099072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698497.258762:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698497.258768:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698497.258768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698497.258774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00080000:0.0:1787698497.258778:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258781:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698497.258784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace90000 x1874531146098432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698497.258786:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698497.258799:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146099200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698497.258805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146098432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698497.258814:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698497.258819:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698497.258819:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698497.258820:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698497.258825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698497.258828:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6396 00000100:00100000:1.0:1787698497.258831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098432:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698497.258832:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698497.258838:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698497.258841:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698497.258848:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146099328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698497.258856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698497.258871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.258881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace93800 x1874531146098560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.258888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146098560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.258893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.258895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.258897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698497.258901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698497.258911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.258920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace93b80 x1874531146098688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.258930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146098688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.258938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.258941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.258944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698497.258950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698497.258966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.258972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace92680 x1874531146098816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.258979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146098816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.258984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.258986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.258987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146098816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698497.258990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146098944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698497.259000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.259005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace93100 x1874531146098944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.259012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146098944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.259016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.259018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.259019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146098944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698497.259022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146099072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698497.259031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.259036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace92300 x1874531146099072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.259042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146099072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.259046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.259047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.259049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146099072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698497.259052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146099200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698497.259060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.259064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace91c00 x1874531146099200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.259070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146099200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.259079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.259080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.259082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146099200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698497.259093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146099328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698497.259101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698497.259106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698497/real 1787698497] req@ffff8f51ace90380 x1874531146099328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698497.259112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146099328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698497.259116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698497.259118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698497.259119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146099328:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698498.154499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698499.178544:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698500.202510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698501.226460:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698502.250493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698502.378438:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378449:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698502.378454:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698502.378465:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378476:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378483:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698502.378520:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146099456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698502.378548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378552:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698502.378554:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698502.378560:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378564:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378568:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378578:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698502.378592:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146099584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698502.378596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146099456:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698502.378611:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698502.378616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698502.378621:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378624:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378634:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698502.378645:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146099712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698502.378654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698502.378659:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698502.378664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698502.378665:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378671:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378676:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378682:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698502.378693:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146099840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698502.378703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378707:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698502.378709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698502.378711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace93480 x1874531146099456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698502.378716:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378729:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698502.378740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146099456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698502.378748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146099968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698502.378754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698502.378763:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698502.378763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698502.378771:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787698502.378771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146099456:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698502.378775:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698502.378782:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378786:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378790:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378794:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698502.378804:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146100096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698502.378810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099584:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698502.378817:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378822:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698502.378823:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698502.378828:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6402 00000800:00000100:1.0:1787698502.378838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.378849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace91500 x1874531146099584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698502.378851:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698502.378860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146099584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698502.378863:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378873:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698502.378876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.378881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.378886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698502.378888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146100224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698502.378896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099712:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698502.378904:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698502.378908:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698502.378910:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698502.378915:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00080000:0.0:1787698502.378919:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698502.378921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698502.378926:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698502.378932:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698502.378935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace91880 x1874531146099712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698502.378946:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146100352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698502.378952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146099712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698502.378961:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698502.378966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698502.378967:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698502.378970:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698502.378974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698502.378978:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6402 00000100:00100000:1.0:1787698502.378982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099712:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698502.378984:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698502.378991:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698502.378995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099840:192.168.201.125@tcp:8:kworker.0 00000008:00100000:0.0:1787698502.378997:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698502.379013:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146100480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698502.379038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.379050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace92d80 x1874531146099840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698502.379060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146099840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698502.379068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.379071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.379073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698502.379079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698502.379096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.379106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace90e00 x1874531146099968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698502.379115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146099968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698502.379138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.379142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.379145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146099968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698502.379162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698502.379181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.379198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace91180 x1874531146100096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698502.379210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146100096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698502.379215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.379217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.379219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698502.379224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698502.379241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.379252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace93480 x1874531146100224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698502.379262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146100224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698502.379268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.379271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.379275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698502.379280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698502.379295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.379304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace90380 x1874531146100352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698502.379315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146100352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698502.379324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.379326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.379327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698502.379331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698502.379342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698502.379348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698502/real 1787698502] req@ffff8f51ace91c00 x1874531146100480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698502.379383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146100480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698502.379388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698502.379390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698502.379392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146100480:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698503.274520:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698504.298520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698505.322550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698506.346503:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698507.370499:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698507.498482:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498492:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698507.498496:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498506:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498515:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498522:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698507.498557:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146100608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698507.498583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498587:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698507.498589:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498594:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498598:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698507.498612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698507.498626:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146100736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698507.498633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146100608:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698507.498637:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698507.498641:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498644:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498647:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498649:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698507.498653:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698507.498662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146100864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698507.498668:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498670:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698507.498671:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498673:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498675:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498677:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698507.498680:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698507.498686:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146100992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698507.498692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698507.498696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698507.498696:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698507.498699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498703:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498706:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498710:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698507.498714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698507.498724:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146101120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698507.498732:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787698507.498742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f518738b100 x1874531146100608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698507.498747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498759:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498762:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498766:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698507.498770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146100608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698507.498772:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698507.498784:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146101248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698507.498785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.498795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698507.498795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698507.498803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698507.498804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146100608:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698507.498807:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498811:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498814:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698507.498816:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698507.498824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146101376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698507.498832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146100736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698507.498835:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498841:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698507.498843:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498847:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00080000:3.0:1787698507.498852:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498856:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698507.498861:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698507.498866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698507.498877:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146101504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698507.498884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f5187388e00 x1874531146100736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698507.498889:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698507.498897:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698507.498901:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698507.498907:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6407 00000100:00100000:1.0:1787698507.498909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146100736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698507.498911:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698507.498919:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698507.498928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698507.498928:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698507.498936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698507.498943:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146101632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698507.498944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146100736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698507.498956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146100864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698507.498979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.498992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f5187389f80 x1874531146100864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146100864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146100864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698507.499018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146100992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698507.499032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.499056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f518738b480 x1874531146100992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146100992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146100992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698507.499084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146101120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698507.499101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.499110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f5187388000 x1874531146101120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146101120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146101120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698507.499138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698507.499154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.499164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f518738a680 x1874531146101248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146101248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698507.499208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698507.499224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.499233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f5187389180 x1874531146101376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146101376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698507.499256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698507.499267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.499273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f518738b100 x1874531146101504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146101504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698507.499293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698507.499301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698507.499306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698507/real 1787698507] req@ffff8f5187389500 x1874531146101632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698507.499312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146101632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698507.499319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698507.499322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698507.499325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146101632:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698508.394481:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698509.418542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698510.442546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698511.466530:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698512.490559:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698512.618516:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618525:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698512.618528:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618536:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618545:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618552:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698512.618584:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146101760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698512.618610:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698512.618614:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618626:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618630:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618635:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618645:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698512.618657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146101760:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698512.618663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146101888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698512.618678:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618682:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698512.618684:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618688:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618692:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618695:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618699:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698512.618710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146102016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698512.618713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698512.618723:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618726:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698512.618727:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618731:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618736:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618738:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618741:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698512.618749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146102144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698512.618755:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618757:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698512.618759:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618762:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618764:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698512.618768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704e680 x1874531146101760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698512.618769:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618774:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698512.618787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146102272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698512.618794:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618797:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698512.618801:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698512.618803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146101760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698512.618806:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618811:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618817:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618823:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698512.618833:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146102400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698512.618837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.618846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698512.618852:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698512.618856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146101760:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698512.618857:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698512.618861:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618866:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618868:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618871:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698512.618881:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146102528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698512.618886:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146101888:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698512.618890:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698512.618896:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698512.618897:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618900:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618903:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618905:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618908:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698512.618916:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146102656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698512.618922:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698512.618928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698512.618928:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698512.618932:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698512.618937:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6412 00000100:00080000:2.0:1787698512.618941:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698512.618944:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698512.618947:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698512.618971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704c000 x1874531146101888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698512.618982:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146102784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698512.618992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146101888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146101888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698512.619018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146102016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698512.619039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704d180 x1874531146102016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146102016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146102016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698512.619081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146102144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698512.619098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704fb80 x1874531146102144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146102144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146102144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698512.619141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146102272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698512.619160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704df80 x1874531146102272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146102272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146102272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698512.619202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698512.619222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704f800 x1874531146102400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146102400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698512.619295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698512.619319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704ce00 x1874531146102528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146102528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698512.619388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698512.619404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704c380 x1874531146102656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146102656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698512.619437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698512.619446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698512.619452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698512/real 1787698512] req@ffff8f518704e680 x1874531146102784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698512.619459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146102784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698512.619464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698512.619466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698512.619468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146102784:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698513.514531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698514.538543:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698515.562538:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698516.586531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698517.610578:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698517.738481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738492:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698517.738496:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738508:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738519:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738526:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698517.738576:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146102912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738604:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738607:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698517.738609:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738612:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738616:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738619:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698517.738638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146103040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738645:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738650:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698517.738651:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738656:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00100000:1.0:1787698517.738657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146102912:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698517.738659:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738665:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698517.738678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146103168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738685:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738687:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698517.738689:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738691:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738694:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738696:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738699:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698517.738707:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146103296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738712:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698517.738716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738719:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738721:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738723:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738727:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698517.738728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698517.738738:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146103424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738746:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738748:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698517.738750:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738752:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738755:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738757:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738760:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698517.738771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704c700 x1874531146102912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698517.738772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146103552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738806:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698517.738809:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698517.738811:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738815:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00100000:1.0:1787698517.738816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146102912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698517.738818:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738824:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698517.738842:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146103680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698517.738845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698517.738854:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698517.738855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698517.738858:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698517.738863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146102912:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698517.738863:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738883:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738887:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698517.738892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103040:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698517.738892:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698517.738901:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698517.738913:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146103808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698517.738920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698517.738925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698517.738925:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698517.738927:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698517.738933:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6417 00000100:00080000:2.0:1787698517.738936:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698517.738939:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698517.738942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704ed80 x1874531146103040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698517.738945:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698517.738959:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146103936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698517.738962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146103040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.738985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.738988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.738991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698517.739012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698517.739024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704dc00 x1874531146103168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146103168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698517.739052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698517.739062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704ea00 x1874531146103296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146103296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698517.739088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698517.739097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704e300 x1874531146103424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146103424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146103424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698517.739130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698517.739144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704d880 x1874531146103552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146103552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698517.739172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698517.739181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704d500 x1874531146103680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146103680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698517.739205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698517.739215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704f480 x1874531146103808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146103808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698517.739244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698517.739253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698517.739258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698517/real 1787698517] req@ffff8f518704e680 x1874531146103936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698517.739265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146103936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698517.739270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698517.739272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698517.739273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146103936:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698518.634500:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698519.658527:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698520.682570:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698521.706500:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698522.730499:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698522.858553:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698522.858563:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698522.858579:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.858590:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:0.0:1787698522.858600:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.858608:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698522.858644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146104064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698522.858672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698522.858676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698522.858678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.858683:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:0.0:1787698522.858687:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.858691:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698522.858700:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698522.858713:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146104192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698522.858725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698522.858729:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698522.858731:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.858736:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:0.0:1787698522.858740:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.858744:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698522.858750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146104064:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698522.858753:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698522.858767:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146104320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698522.858808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698522.858813:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698522.858815:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.858820:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:0.0:1787698522.858825:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698522.858828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698522.858833:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698522.858841:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698522.858856:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146104448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698522.858867:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698522.858872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace92300 x1874531146104064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.858874:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698522.858879:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.858889:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00100000:1.0:1787698522.858894:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146104064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.858895:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.858906:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698522.858908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698522.858913:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698522.858914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.858923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146104064:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698522.858934:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146104576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698522.858935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104192:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698522.858952:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698522.858957:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698522.858959:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.858966:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6422 00000800:00000100:1.0:1787698522.858970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698522.858973:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.858980:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698522.858987:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698522.858990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace93100 x1874531146104192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698522.859003:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146104704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698522.859009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146104192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.859030:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698522.859031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698522.859041:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698522.859042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104192:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698522.859047:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.859054:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:0.0:1787698522.859059:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.859064:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104320:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698522.859072:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698522.859090:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146104832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698522.859091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace92680 x1874531146104320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.859109:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698522.859115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146104320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.859117:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698522.859120:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698522.859125:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:0.0:1787698522.859132:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698522.859142:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698522.859143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787698522.859151:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698522.859153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698522.859162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698522.859171:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146104960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698522.859180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698522.859189:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698522.859191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace93b80 x1874531146104448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.859201:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698522.859206:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698522.859208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146104448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698522.859214:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6422 00000100:00080000:1.0:1787698522.859218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698522.859222:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698522.859224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.859229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104448:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698522.859233:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104576:192.168.201.125@tcp:8:kworker.0 00000008:00100000:0.0:1787698522.859244:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698522.859253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698522.859263:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146105088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698522.859264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace93800 x1874531146104576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698522.859299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146104576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698522.859308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.859313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146104576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698522.859316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146104704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698522.859328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace90000 x1874531146104704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698522.859346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146104704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698522.859377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.859382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146104704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698522.859393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146104832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698522.859406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace90700 x1874531146104832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698522.859422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146104832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698522.859427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.859431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146104832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698522.859434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146104960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698522.859444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace93100 x1874531146104960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698522.859456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146104960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698522.859465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.859469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146104960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698522.859473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146105088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698522.859481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698522.859487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698522] req@ffff8f51ace92300 x1874531146105088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698522.859494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146105088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698522.859499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698522.859501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698522.859502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146105088:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698523.754511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698524.778629:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698525.802550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698526.826550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698527.850616:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698527.978499:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.978511:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698527.978517:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698527.978529:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.978551:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.978559:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698527.978611:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146105216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698527.978652:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.978656:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698527.978658:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698527.978723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146105216:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698527.978763:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.978777:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.978785:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698527.978808:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698527.978819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698527.978848:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146105344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698527.978869:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.978875:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698527.978878:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698527.978884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace91c00 x1874531146105216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698527.978886:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.978895:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.978904:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698527.978922:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698527.978945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146105216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698527.978946:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146105472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698527.979001:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.979007:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698527.979008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698527.979011:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698527.979019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698527.979022:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.979031:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698527.979032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146105216:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698527.979042:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698527.979051:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698527.979076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698527.979082:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146105600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698527.979104:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.979108:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698527.979111:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698527.979118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698527.979119:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.979129:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.979135:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698527.979145:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698527.979145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace90380 x1874531146105344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698527.979176:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146105728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698527.979179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146105344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.979235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698527.979238:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698527.979244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698527.979247:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698527.979253:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698527.979259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105344:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698527.979264:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.979275:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.979282:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698527.979288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105472:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698527.979294:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698527.979318:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146105856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698527.979327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698527.979342:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.979346:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698527.979348:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698527.979402:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:0.0:1787698527.979407:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.979411:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698527.979417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace93480 x1874531146105472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698527.979418:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698527.979442:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146105984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698527.979452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146105472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698527.979462:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.979471:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698527.979475:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698527.979486:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00080000:1.0:1787698527.979489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.979498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698527.979503:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698527.979509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105472:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698527.979517:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698527.979526:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698527.979537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698527.979554:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146106112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698527.979572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698527.979585:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698527.979589:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698527.979591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698527.979597:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6427 00000100:00100000:1.0:1787698527.979601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace91180 x1874531146105600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698527.979603:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698527.979612:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698527.979619:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698527.979635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146105600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698527.979636:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146106240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698527.979667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.979675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698527.979681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698527.979705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698527.979736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698527.979764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace90e00 x1874531146105728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698527.979780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146105728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.979789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.979793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698527.979797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146105728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698527.979804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146105856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698527.979823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698527.979833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace92d80 x1874531146105856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698527.979853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146105856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.979862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.979865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698527.979868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146105856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698527.979875:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146105984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698527.979893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698527.979903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace91880 x1874531146105984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698527.979915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146105984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.979923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.979926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698527.979929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146105984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698527.979948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146106112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698527.979965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698527.979975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace91500 x1874531146106112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698527.979987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146106112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.980001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.980005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698527.980008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146106112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698527.980015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146106240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698527.980031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698527.980041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff8f51ace93480 x1874531146106240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698527.980052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146106240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.980060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698527.980064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698527.980068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146106240:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698528.874532:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698529.898546:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698530.922563:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698531.946528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698532.970534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698533.098500:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.098512:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698533.098516:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.098528:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.098539:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.098547:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698533.098595:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146106368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698533.098634:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.098638:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698533.098640:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.098644:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.098648:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.098653:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.098663:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698533.098682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146106368:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698533.098685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146106496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698533.098701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.098704:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698533.098706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.098710:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.098713:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.098717:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.098722:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698533.098734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146106624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698533.098750:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698533.098757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698533.098758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698533.098763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.098768:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.098772:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.098776:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.098782:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698533.098793:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146106752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698533.098802:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.098805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698533.098808:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.098813:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00100000:1.0:1787698533.098814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f518738bb80 x1874531146106368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698533.098818:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.098827:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.098835:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698533.098846:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146106880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698533.098856:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698533.098858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146106368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698533.098862:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698533.098867:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.098878:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.098886:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698533.098891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698533.098894:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698533.098900:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698533.098903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698533.098908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146106368:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698533.098919:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146107008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698533.098929:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.098932:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698533.098934:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698533.098941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106496:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698533.098941:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.098944:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.098951:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.098957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698533.098971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698533.098973:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146107136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698533.098985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f5187389c00 x1874531146106496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698533.098985:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.098992:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698533.098996:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.099003:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00080000:3.0:1787698533.099007:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698533.099009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146106496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698533.099013:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.099023:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698533.099025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698533.099041:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146107264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698533.099042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698533.099066:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698533.099071:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698533.099073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698533.099074:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698533.099082:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6432 00000100:00100000:1.0:1787698533.099086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f518738a300 x1874531146106624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698533.099088:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698533.099097:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698533.099104:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698533.099114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146106624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698533.099118:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146107392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698533.099134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698533.099161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698533.099187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698533.099200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f518738ad80 x1874531146106752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698533.099215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146106752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698533.099224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698533.099237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698533.099258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698533.099271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f518738b800 x1874531146106880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698533.099286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146106880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698533.099294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146106880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698533.099309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698533.099331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698533.099343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f5187388a80 x1874531146107008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698533.099392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146107008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698533.099402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698533.099431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698533.099463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698533.099477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f5187388380 x1874531146107136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698533.099491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146107136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698533.099499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698533.099513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698533.099533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698533.099545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f518738aa00 x1874531146107264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698533.099558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146107264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698533.099572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698533.099585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698533.099622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698533.099635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698533/real 1787698533] req@ffff8f518738bb80 x1874531146107392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698533.099648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146107392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698533.099656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698533.099659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698533.099663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146107392:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698533.994532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698535.018523:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698536.042572:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698537.066628:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698538.090532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698538.218412:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218420:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698538.218430:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218438:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218447:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218453:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698538.218484:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146107520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698538.218508:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218511:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698538.218513:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218538:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218542:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218546:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698538.218555:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698538.218567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146107520:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698538.218570:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146107648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698538.218585:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218590:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698538.218592:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218601:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218606:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698538.218612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698538.218626:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146107776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698538.218635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698538.218636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218641:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698538.218642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218646:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218649:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218652:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698538.218657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698538.218670:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146107904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698538.218679:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698538.218685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218690:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218694:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698538.218695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f5187389500 x1874531146107520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698538.218703:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698538.218711:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698538.218718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146107520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698538.218727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146108032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698538.218735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.218743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698538.218747:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698538.218748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146107520:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698538.218756:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698538.218760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218765:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218770:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218775:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698538.218780:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698538.218782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146107648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698538.218797:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146108160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698538.218811:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698538.218818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698538.218818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698538.218824:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218829:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218834:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698538.218840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f518738b100 x1874531146107648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698538.218853:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698538.218859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146107648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698538.218863:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698538.218876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.218883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698538.218884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146108288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698538.218889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146107648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698538.218904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146107776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698538.218907:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218913:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698538.218915:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218920:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00080000:3.0:1787698538.218925:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218930:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698538.218935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698538.218938:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698538.218951:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146108416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698538.218956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f5187389180 x1874531146107776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698538.218962:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698538.218971:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698538.218975:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698538.218983:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6437 00000100:00100000:1.0:1787698538.218986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146107776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698538.218988:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698538.218999:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698538.219004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698538.219007:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698538.219011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146107776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698538.219026:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146108544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698538.219028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146107904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698538.219061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698538.219076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f518738a680 x1874531146107904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698538.219089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146107904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698538.219097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.219101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146107904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698538.219111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146108032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698538.219130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698538.219143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f5187388000 x1874531146108032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698538.219155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146108032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698538.219164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.219167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146108032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698538.219188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698538.219210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698538.219223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f518738b480 x1874531146108160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698538.219235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146108160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698538.219245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.219248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698538.219258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698538.219277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698538.219289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f5187389f80 x1874531146108288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698538.219302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146108288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698538.219310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.219313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698538.219322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698538.219341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698538.219352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f5187388e00 x1874531146108416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698538.219405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146108416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698538.219419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.219429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698538.219439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698538.219458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698538.219470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698538/real 1787698538] req@ffff8f5187389500 x1874531146108544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698538.219482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146108544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698538.219490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698538.219493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698538.219497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146108544:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698539.114560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698540.138511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698541.162553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698542.186567:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698543.210577:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698543.338425:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338448:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698543.338452:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338463:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338474:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338482:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698543.338518:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146108672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698543.338548:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338551:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698543.338552:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338556:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338559:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338561:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338569:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698543.338578:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146108800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698543.338587:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698543.338590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146108672:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698543.338591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698543.338594:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338601:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338603:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338606:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338609:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698543.338619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146108928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698543.338631:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338633:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698543.338634:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338637:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338639:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338642:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338645:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698543.338651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698543.338654:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146109056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698543.338662:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338665:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698543.338666:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338670:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338673:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338675:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338677:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698543.338685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146109184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698543.338692:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698543.338695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f518738bb80 x1874531146108672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698543.338696:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698543.338699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338705:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338709:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338713:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698543.338728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146109312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698543.338733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146108672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698543.338737:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338744:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698543.338747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338754:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338758:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698543.338759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698543.338760:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698543.338771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698543.338777:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146109440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698543.338778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146108672:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698543.338792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338794:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698543.338796:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698543.338799:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00080000:3.0:1787698543.338801:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698543.338803:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338806:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698543.338810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146108800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698543.338816:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146109568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698543.338829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698543.338832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698543.338834:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698543.338837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698543.338842:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6442 00000100:00100000:1.0:1787698543.338852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f518738aa00 x1874531146108800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698543.338856:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698543.338865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146108800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698543.338866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698543.338879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698543.338880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.338885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.338889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146108800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698543.338895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146108928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698543.338900:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146109696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698543.338911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.338923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f5187388380 x1874531146108928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.338934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146108928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.338939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.338941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.338943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146108928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698543.338947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146109056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698543.338958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.338964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f5187388a80 x1874531146109056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.338971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146109056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.338976:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.338978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.338982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146109056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698543.338988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146109184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698543.339006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.339017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f518738b800 x1874531146109184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.339030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146109184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.339039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.339041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.339044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146109184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698543.339048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698543.339063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.339071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f518738ad80 x1874531146109312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.339084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146109312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.339090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.339093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.339096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698543.339102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698543.339120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.339132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f518738a300 x1874531146109440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.339146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146109440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.339153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.339155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.339157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698543.339173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698543.339188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.339195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f5187389c00 x1874531146109568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.339202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146109568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.339212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.339214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.339216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698543.339229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698543.339240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698543.339247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698543/real 1787698543] req@ffff8f518738bb80 x1874531146109696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698543.339254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146109696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698543.339259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698543.339262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698543.339264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146109696:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698544.234641:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698545.258500:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698546.282542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698547.306587:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698548.330519:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698548.458453:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458459:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698548.458462:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458470:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458479:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458484:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698548.458514:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146109824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458535:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458537:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698548.458538:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458542:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458543:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458546:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458552:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698548.458560:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146109952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458567:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458569:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698548.458570:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458572:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458574:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458576:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458580:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698548.458585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146109824:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698548.458595:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146110080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458609:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698548.458611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458615:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458619:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458622:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698548.458637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146110208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458643:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698548.458647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458649:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458652:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698548.458663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146109952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698548.458667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146110336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458674:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458676:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698548.458677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698548.458693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146110080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698548.458712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146110208:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698548.458723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458727:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698548.458729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146110336:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698548.458732:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458739:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698548.458749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146110464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698548.458761:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458764:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00100000:1.0:1787698548.458767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698548.458768:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458773:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458777:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698548.458786:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146110592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458794:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698548.458795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458798:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458801:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698548.458804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698548.458805:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458810:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698548.458819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146110720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698548.458826:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698548.458828:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698548.458829:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698548.458831:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6448 00000100:00080000:3.0:1787698548.458834:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698548.458837:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698548.458841:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698548.458843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698548.458855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146110848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698548.458886:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110848:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698549.354539:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698550.378498:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698551.402531:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698551.530581:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787698551.530623:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698551.530639:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530688:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530703:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530722:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530731:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530741:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530753:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698551.530755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f518738bb80 x1874531146110848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698551.530763:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698551.530776:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787698551.530790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146110848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.530804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.530809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.530814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698551.530821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f518738b100 x1874531146110720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.530833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146110720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.530851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.530855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.530858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698551.530866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f5187389180 x1874531146110592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.530877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146110592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.530884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.530887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.530890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698551.530896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f518738a680 x1874531146110464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.530906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146110464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.530914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.530917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.530920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146110464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698551.530926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f5187389f80 x1874531146110080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.530936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146110080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.530943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.530946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.530949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146110080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698551.530955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f518738b480 x1874531146110208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.530965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146110208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.530971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.530975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.530978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146110208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698551.530984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f5187388000 x1874531146110336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.530994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146110336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.531002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.531005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.531008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146110336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698551.531014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f5187389500 x1874531146109824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.531024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146109824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.531031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.531034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.531037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146109824:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698551.531043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698548/real 1787698551] req@ffff8f5187388e00 x1874531146109952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698551.531053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146109952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698551.531060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698551.531063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698551.531065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146109952:192.168.201.125@tcp:38:kworker.0 00010000:00010000:0.0:1787698552.426536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698553.450538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698553.578525:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578533:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698553.578536:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578543:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578561:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578566:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698553.578596:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146110976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578616:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698553.578617:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578631:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578635:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578639:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578658:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698553.578661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146110976:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698553.578674:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146111104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578684:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578687:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698553.578688:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578692:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578695:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578699:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578714:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698553.578726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146111232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578734:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578738:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787698553.578741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698553.578741:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578749:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578753:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578757:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698553.578775:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146111360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578784:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578787:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698553.578791:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698553.578792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704c380 x1874531146110976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698553.578799:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578806:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578813:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578819:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698553.578827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146110976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698553.578832:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146111488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698553.578846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.578856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698553.578860:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698553.578864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146110976:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698553.578866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698553.578870:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578877:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578880:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578884:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578889:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698553.578900:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146111616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578909:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578913:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698553.578926:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578930:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578934:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578938:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578942:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698553.578952:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146111744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578960:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578963:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698553.578964:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.578967:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.578971:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.578974:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.578977:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698553.578986:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146111872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698553.578995:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698553.578998:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698553.578999:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698553.579003:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6453 00000100:00080000:2.0:1787698553.579006:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698553.579009:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698553.579014:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698553.579026:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146112000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698553.580474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698553.580532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704ce00 x1874531146111104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146111104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698553.580606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698553.580625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704f800 x1874531146111232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146111232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698553.580665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698553.580684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704df80 x1874531146111360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146111360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698553.580731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698553.580751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704fb80 x1874531146111488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146111488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146111488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698553.580790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146111616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698553.580809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704d180 x1874531146111616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146111616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146111616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698553.580845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146111744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698553.580858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704c000 x1874531146111744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146111744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146111744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698553.580883:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146111872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698553.580898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704c380 x1874531146111872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146111872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146111872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698553.580935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146112000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698553.580952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698553.580962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698553/real 1787698553] req@ffff8f518704e680 x1874531146112000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698553.580977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146112000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698553.580983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698553.580986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698553.580989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146112000:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698554.474512:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698555.498539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698556.522590:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698557.546559:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698558.570525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698558.698437:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698445:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698558.698448:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698455:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698464:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698469:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698558.698498:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146112128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698519:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698524:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698558.698525:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698528:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698530:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698532:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698539:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698558.698548:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146112256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698555:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698557:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698558.698558:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698560:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698562:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698565:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698567:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698558.698573:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146112384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698580:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698558.698581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146112128:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698558.698582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698558.698585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698591:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698593:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698595:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698607:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698558.698622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146112512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698632:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698635:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698558.698637:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698641:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698644:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698648:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698558.698652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698558.698664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146112640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698670:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698672:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698558.698674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698678:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698681:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698684:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698689:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698558.698698:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146112768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698558.698708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704f480 x1874531146112128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698558.698708:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698712:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698558.698715:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698724:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698729:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698558.698733:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698737:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698558.698742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146112128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698558.698747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146112896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698558.698766:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698558.698767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698558.698771:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:1.0:1787698558.698776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698558.698777:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698558.698782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146112128:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698558.698785:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698793:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698558.698798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698558.698808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146113024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698558.698817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698558.698822:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698558.698823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698558.698825:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698558.698831:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6458 00000100:00080000:2.0:1787698558.698836:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698558.698839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704d500 x1874531146112256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698558.698842:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698558.698850:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698558.698868:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146113152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698558.698870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146112256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.698899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.698904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.698908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698558.698930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698558.698948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.698964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704d880 x1874531146112384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.698975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146112384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.698982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.698985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.698988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698558.698994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698558.699014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.699024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704e300 x1874531146112512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.699035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146112512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.699047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.699050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.699053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698558.699059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698558.699077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.699087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704ea00 x1874531146112640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.699098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146112640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.699104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.699107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.699110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146112640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698558.699115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146112768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698558.699131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.699150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704dc00 x1874531146112768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.699161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146112768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.699183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.699186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.699190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146112768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698558.699195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146112896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698558.699212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.699221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704ed80 x1874531146112896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.699232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146112896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.699246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.699248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.699252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146112896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698558.699257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146113024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698558.699273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.699281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704c700 x1874531146113024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.699291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146113024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.699298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.699302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.699304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146113024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698558.699317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146113152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698558.699332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698558.699341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698558/real 1787698558] req@ffff8f518704e680 x1874531146113152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698558.699378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146113152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698558.699388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698558.699391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698558.699394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146113152:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698559.594479:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698560.618515:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698561.642518:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698562.666509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698563.690535:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698563.818453:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818460:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698563.818463:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818471:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818480:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818485:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698563.818514:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146113280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698563.818534:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818537:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698563.818538:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818541:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818542:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818544:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818551:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698563.818577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146113280:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698563.818600:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146113408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698563.818608:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818610:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698563.818611:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818614:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818617:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818619:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698563.818629:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146113536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698563.818635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818637:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698563.818638:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698563.818642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698563.818645:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818648:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818650:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818654:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698563.818661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146113664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698563.818667:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698563.818671:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818673:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818676:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698563.818685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146113792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698563.818691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704c380 x1874531146113280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698563.818693:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698563.818700:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818707:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818711:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818714:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698563.818723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146113280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698563.818725:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146113920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698563.818742:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698563.818744:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818750:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:1.0:1787698563.818754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698563.818754:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818762:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:1.0:1787698563.818763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698563.818769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698563.818771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146113280:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698563.818783:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146114048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698563.818795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698563.818797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113408:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698563.818799:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698563.818803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818806:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818809:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818812:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818817:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698563.818821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698563.818828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146114176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698563.818834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704c000 x1874531146113408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698563.818837:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698563.818844:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698563.818847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698563.818852:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6463 00000100:00080000:2.0:1787698563.818857:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698563.818861:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698563.818867:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698563.818871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146113408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.818878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.818880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698563.818883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146114304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698563.818885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698563.818895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698563.818911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.818922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704d180 x1874531146113536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.818933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146113536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.818940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.818943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.818946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698563.818951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698563.818967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.818973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704fb80 x1874531146113664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.818980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146113664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.818993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.818996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.818999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698563.819004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698563.819021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.819030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704df80 x1874531146113792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.819037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146113792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.819042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.819044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.819046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146113792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698563.819049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146113920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698563.819060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.819066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704f800 x1874531146113920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.819072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146113920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.819098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.819101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.819104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146113920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698563.819116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146114048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698563.819128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.819135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704ce00 x1874531146114048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.819144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146114048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.819150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.819153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.819156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146114048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698563.819161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146114176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698563.819179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.819189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704c380 x1874531146114176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.819198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146114176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.819205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.819208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.819211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146114176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698563.819216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146114304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698563.819233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698563.819243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698563/real 1787698563] req@ffff8f518704e680 x1874531146114304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698563.819250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146114304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698563.819255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698563.819257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698563.819259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146114304:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698564.714478:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698565.738558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698566.762496:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698567.786532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698568.810524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698568.938461:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698568.938472:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698568.938476:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938486:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938506:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.938514:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698568.938550:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146114432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698568.938574:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698568.938578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698568.938580:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938584:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938587:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.938600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938614:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698568.938629:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146114560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698568.938635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146114432:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698568.938646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698568.938658:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698568.938661:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938665:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938669:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.938673:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938678:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698568.938693:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146114688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698568.938700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698568.938712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698568.938714:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698568.938716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938720:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938723:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.938725:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938729:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698568.938738:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146114816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698568.938743:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698568.938746:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698568.938747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938752:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00100000:1.0:1787698568.938753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f5187389c00 x1874531146114432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698568.938754:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.938759:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938765:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698568.938776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146114944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698568.938783:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698568.938784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146114432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698568.938787:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698568.938791:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698568.938801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.938804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.938807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146114432:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698568.938810:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938816:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.938820:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938825:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698568.938829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698568.938840:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146115072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698568.938849:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698568.938851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698568.938854:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698568.938857:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938861:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938864:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698568.938865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f518738a300 x1874531146114560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698568.938869:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938877:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698568.938889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146114560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.938896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.938898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698568.938902:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146115200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698568.938903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698568.938916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698568.938918:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698568.938927:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698568.938930:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698568.938934:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.938939:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698568.938941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698568.938946:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.938952:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698568.938957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f518738ad80 x1874531146114688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698568.938966:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146115328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698568.938975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146114688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698568.939008:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698568.939013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698568.939017:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698568.939018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698568.939021:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698568.939025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114816:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698568.939028:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6468 00000100:00080000:3.0:1787698568.939034:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698568.939039:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698568.939045:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698568.939048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698568.939061:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146115456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698568.939062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f518738b800 x1874531146114816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698568.939093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146114816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.939112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.939115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698568.939118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698568.939130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698568.939136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f5187388a80 x1874531146114944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698568.939144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146114944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.939154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.939156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146114944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698568.939172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698568.939191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698568.939202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f5187388380 x1874531146115072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698568.939213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146115072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.939223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.939227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698568.939232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698568.939249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698568.939259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f518738aa00 x1874531146115200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698568.939270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146115200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.939280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.939283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698568.939289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698568.939317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698568.939327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f5187389c00 x1874531146115328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698568.939338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146115328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.939349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.939376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698568.939384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698568.939414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698568.939426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff8f5187388e00 x1874531146115456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698568.939437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146115456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.939445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698568.939447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698568.939450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146115456:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698569.834542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698570.858522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698571.882533:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698572.906505:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698573.930512:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698574.058475:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058484:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698574.058488:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058496:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058504:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058509:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698574.058537:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146115584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698574.058558:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058562:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698574.058564:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058568:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058572:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058577:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698574.058603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146115584:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698574.058604:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146115712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698574.058620:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058623:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698574.058626:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058637:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058642:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058645:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058650:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698574.058664:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146115840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698574.058665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698574.058674:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698574.058678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058681:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058684:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058686:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058689:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698574.058696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146115968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698574.058705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698574.058708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058712:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058714:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698574.058718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704c700 x1874531146115584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698574.058725:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146116096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698574.058741:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698574.058745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058749:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00100000:1.0:1787698574.058752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146115584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698574.058752:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058757:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698574.058772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698574.058772:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146116224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698574.058781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698574.058783:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698574.058786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698574.058790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146115584:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698574.058790:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058797:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058800:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058802:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058804:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698574.058810:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146116352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698574.058817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698574.058819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146115712:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698574.058820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698574.058823:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058828:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058830:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058832:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058835:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698574.058843:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146116480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698574.058849:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698574.058853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698574.058854:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698574.058858:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698574.058862:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6473 00000100:00080000:2.0:1787698574.058864:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698574.058866:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698574.058869:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698574.058872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704ed80 x1874531146115712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698574.058878:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146116608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698574.058928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146115712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.058939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.058942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.058946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146115712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698574.058952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146115840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698574.058971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.058981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704dc00 x1874531146115840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.058992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146115840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.058998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146115840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698574.059009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146115968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698574.059023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.059031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704ea00 x1874531146115968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.059041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146115968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.059052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146115968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698574.059061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146116096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698574.059074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.059082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704e300 x1874531146116096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.059092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146116096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.059099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146116096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698574.059109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698574.059122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.059131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704d880 x1874531146116224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.059141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146116224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.059147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698574.059169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698574.059183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.059192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704d500 x1874531146116352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.059201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146116352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.059208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698574.059219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698574.059240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.059248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704f480 x1874531146116480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.059258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146116480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.059265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698574.059277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698574.059291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698574.059299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698574/real 1787698574] req@ffff8f518704c700 x1874531146116608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698574.059308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146116608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698574.059315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698574.059318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698574.059321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146116608:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698574.954548:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698575.978538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698577.002504:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698578.026520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698579.050497:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698579.178445:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178453:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698579.178457:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178464:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178475:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178481:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698579.178517:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146116736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698579.178543:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178546:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698579.178547:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178550:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178553:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178555:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178562:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698579.178573:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146116864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698579.178580:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178583:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698579.178584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146116736:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698579.178585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178601:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698579.178607:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146116992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698579.178612:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178614:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698579.178615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178618:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178620:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178622:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178625:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698579.178631:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146117120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698579.178637:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178639:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698579.178640:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178643:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178644:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178646:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178649:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698579.178653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698579.178657:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146117248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698579.178663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178666:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698579.178667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178671:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178675:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178678:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178683:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698579.178694:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146117376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698579.178702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704e680 x1874531146116736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698579.178707:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178716:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698579.178717:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178720:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178722:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698579.178735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146116736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698579.178740:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146117504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698579.178755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698579.178756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698579.178762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698579.178764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698579.178765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698579.178773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146116736:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698579.178774:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178778:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178782:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178784:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698579.178790:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146117632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698579.178797:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698579.178799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146116864:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698579.178800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698579.178804:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698579.178808:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6478 00000100:00080000:2.0:1787698579.178810:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698579.178812:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698579.178829:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698579.178837:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146117760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698579.178850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.178863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704c380 x1874531146116864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.178881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146116864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.178890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.178892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.178896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146116864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698579.178901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146116992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698579.178920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.178932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704ce00 x1874531146116992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.178943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146116992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.178950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.178951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.178954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146116992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698579.178957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146117120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698579.178971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.178978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704f800 x1874531146117120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.178985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146117120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.178998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.178999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.179001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146117120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698579.179004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146117248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698579.179014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.179020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704df80 x1874531146117248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.179026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146117248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.179030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.179032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.179034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146117248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698579.179036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698579.179045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.179049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704fb80 x1874531146117376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.179055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146117376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.179060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.179061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.179064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698579.179067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698579.179075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.179080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704d180 x1874531146117504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.179086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146117504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.179090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.179091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.179093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698579.179105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698579.179113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.179119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704c000 x1874531146117632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.179125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146117632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.179129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.179131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.179132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698579.179135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698579.179144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698579.179149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698579/real 1787698579] req@ffff8f518704e680 x1874531146117760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698579.179155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146117760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698579.179159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698579.179161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698579.179162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146117760:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698580.074529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698581.098489:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698582.122551:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698583.146493:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698584.170510:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698584.298422:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298429:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698584.298433:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298441:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298449:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298454:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698584.298484:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146117888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698584.298506:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298509:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698584.298510:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298513:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298515:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298518:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298529:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698584.298538:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146118016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698584.298550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298552:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698584.298552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146117888:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698584.298555:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298561:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298563:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298565:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298568:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698584.298576:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146118144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698584.298582:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698584.298585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298587:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298588:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298590:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298593:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698584.298599:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146118272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698584.298604:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298606:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698584.298607:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298609:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298612:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298616:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698584.298616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298622:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698584.298630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146118400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698584.298635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298638:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698584.298639:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298644:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298647:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298651:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298656:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698584.298657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace90380 x1874531146117888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698584.298671:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146118528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698584.298674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146117888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698584.298694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698584.298697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698584.298698:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698584.298701:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698584.298705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698584.298708:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00100000:1.0:1787698584.298712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146117888:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698584.298713:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698584.298750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698584.298754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146118656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698584.298765:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698584.298768:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298770:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298774:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298779:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:1.0:1787698584.298780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298782:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698584.298791:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146118784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698584.298796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace91c00 x1874531146118016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698584.298801:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698584.298803:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698584.298804:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698584.298809:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6483 00000100:00080000:0.0:1787698584.298812:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698584.298816:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698584.298821:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698584.298830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146118912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698584.298833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146118016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.298849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.298854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.298858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698584.298865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698584.298884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.298892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace92300 x1874531146118144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.298900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146118144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.298905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.298907:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.298909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698584.298912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698584.298923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.298928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace93100 x1874531146118272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.298934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146118272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.298943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.298945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.298946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698584.298950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698584.298963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.298972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace90700 x1874531146118400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.298984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146118400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.298991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.298994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.298997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146118400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698584.299003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698584.299023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.299034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace90000 x1874531146118528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.299045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146118528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.299053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.299056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.299060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698584.299077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698584.299091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.299098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace93800 x1874531146118656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.299106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146118656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.299110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.299112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.299114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698584.299117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698584.299125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.299130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace93b80 x1874531146118784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.299136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146118784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.299140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.299142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.299143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698584.299146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698584.299162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698584.299168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698584/real 1787698584] req@ffff8f51ace92680 x1874531146118912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698584.299174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146118912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698584.299179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698584.299182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698584.299185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146118912:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698585.194528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698586.218573:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698587.242518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698588.266549:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698589.290502:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698589.418469:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418479:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698589.418483:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698589.418492:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418502:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418507:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698589.418542:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146119040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698589.418574:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418578:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698589.418580:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698589.418584:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418588:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418592:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418601:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698589.418616:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146119168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698589.418625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146119040:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698589.418629:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418633:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698589.418635:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698589.418639:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418659:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418664:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698589.418675:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146119296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698589.418682:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418684:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698589.418685:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698589.418690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698589.418692:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418696:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418699:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418703:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698589.418710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146119424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698589.418716:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418719:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698589.418720:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698589.418722:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418725:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698589.418727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704c700 x1874531146119040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698589.418731:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698589.418748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146119040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698589.418753:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146119552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698589.418759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.418766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.418771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146119040:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698589.418771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698589.418783:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698589.418788:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418792:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698589.418796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119168:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698589.418798:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418805:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698589.418820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698589.418821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146119680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698589.418830:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698589.418833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704f480 x1874531146119168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698589.418834:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698589.418837:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698589.418844:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418849:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418856:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418862:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698589.418867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146119168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698589.418872:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146119808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698589.418875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.418884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698589.418888:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698589.418890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119168:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698589.418892:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698589.418895:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698589.418901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119296:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698589.418901:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418905:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418909:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418912:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698589.418920:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146119936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698589.418925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698589.418933:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698589.418935:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698589.418937:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698589.418938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704d500 x1874531146119296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698589.418941:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6489 00000100:00080000:2.0:1787698589.418943:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698589.418945:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698589.418951:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698589.418960:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146120064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698589.418963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146119296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.418978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.418981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.418983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698589.418986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698589.418999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698589.419005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704d880 x1874531146119424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698589.419011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146119424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.419020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.419022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.419024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698589.419026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698589.419035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698589.419039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704e300 x1874531146119552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698589.419045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146119552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.419050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.419052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.419053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146119552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698589.419063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146119680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698589.419072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698589.419077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704ea00 x1874531146119680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698589.419083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146119680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.419088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.419090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.419092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146119680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698589.419095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146119808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698589.419103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698589.419108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704dc00 x1874531146119808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698589.419114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146119808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.419118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.419120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.419121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146119808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698589.419124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146119936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698589.419132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698589.419136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704ed80 x1874531146119936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698589.419142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146119936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.419146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.419149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.419152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146119936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698589.419157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146120064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698589.419173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698589.419183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698589/real 1787698589] req@ffff8f518704c700 x1874531146120064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698589.419195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146120064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698589.419202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698589.419204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698589.419206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146120064:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698590.314506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698591.338518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698592.362491:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698593.386470:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698594.410591:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698594.538512:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538529:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698594.538536:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538550:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538580:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538590:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698594.538638:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146120192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698594.538672:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538676:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698594.538678:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538683:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538687:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538691:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538703:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698594.538716:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146120320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698594.538719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146120192:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698594.538726:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698594.538731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538734:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538736:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538738:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538741:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698594.538751:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146120448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698594.538758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538760:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698594.538761:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538763:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538766:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538769:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538772:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698594.538778:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146120576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698594.538784:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698594.538788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538791:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698594.538798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698594.538799:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538805:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698594.538813:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146120704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698594.538820:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538823:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698594.538824:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538826:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538828:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538830:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538833:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698594.538841:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146120832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698594.538846:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538850:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698594.538852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f5187389500 x1874531146120192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698594.538852:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538859:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538864:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538869:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538874:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698594.538879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146120192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698594.538897:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146120960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698594.538898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.538907:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698594.538914:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698594.538915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146120192:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698594.538919:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698594.538923:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698594.538929:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538933:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538938:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538942:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698594.538949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698594.538957:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146121088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698594.538969:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698594.538972:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698594.538975:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698594.538979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698594.538982:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6494 00000100:00080000:3.0:1787698594.538987:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698594.538992:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698594.538997:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698594.538999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f5187388000 x1874531146120320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698594.539011:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146121216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698594.539023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146120320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698594.539066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698594.539088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f518738b480 x1874531146120448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146120448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698594.539139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698594.539161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f5187389f80 x1874531146120576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146120576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698594.539208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698594.539228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f518738a680 x1874531146120704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146120704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146120704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698594.539268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146120832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698594.539286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f5187389180 x1874531146120832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146120832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146120832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698594.539330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146120960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698594.539344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f518738b100 x1874531146120960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146120960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146120960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698594.539431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146121088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698594.539448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f518738bb80 x1874531146121088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146121088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146121088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698594.539478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146121216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698594.539490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698594.539497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698594/real 1787698594] req@ffff8f5187389500 x1874531146121216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698594.539504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146121216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698594.539508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698594.539510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698594.539511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146121216:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698595.434569:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698596.458489:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698597.482540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698598.506568:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698599.530583:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698599.658538:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658553:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698599.658559:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658573:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658587:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658596:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698599.658652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146121344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.658683:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698599.658695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658702:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658706:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658711:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.658723:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698599.658737:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146121472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.658747:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658750:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698599.658752:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658756:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658759:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658763:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.658768:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698599.658778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146121344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698599.658784:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146121600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.658804:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658808:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698599.658809:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658813:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658817:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658822:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.658826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698599.658839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146121728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.658848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658852:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698599.658854:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658857:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658876:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658880:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.658885:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698599.658893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698599.658904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146121856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.658914:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658916:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698599.658920:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658924:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658927:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658931:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.658935:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698599.658943:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146121984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.658949:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.658952:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698599.658953:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.658956:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.658959:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.658966:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698599.658966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace92680 x1874531146121344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698599.658975:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698599.658988:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146122112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.659011:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.659014:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698599.659015:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698599.659023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146121344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698599.659024:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.659032:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698599.659037:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.659041:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698599.659055:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146122240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698599.659062:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698599.659064:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698599.659065:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698599.659069:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6499 00000100:00080000:0.0:1787698599.659071:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698599.659073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698599.659074:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698599.659085:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698599.659091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146121344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698599.659103:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146122368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698599.659108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698599.659162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace93b80 x1874531146121472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146121472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698599.659239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698599.659257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace93800 x1874531146121600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146121600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698599.659318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698599.659342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace90000 x1874531146121728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146121728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698599.659463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698599.659505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace90700 x1874531146121856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146121856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146121856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698599.659564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146121984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698599.659583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace93100 x1874531146121984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146121984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146121984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698599.659628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146122112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698599.659643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace92300 x1874531146122112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146122112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146122112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698599.659687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146122240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698599.659706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659717:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace91c00 x1874531146122240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146122240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146122240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698599.659751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146122368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698599.659771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698599.659784:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698599/real 1787698599] req@ffff8f51ace90380 x1874531146122368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698599.659797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146122368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698599.659806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698599.659809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698599.659813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146122368:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698600.554541:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698601.578581:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698602.602587:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698603.626609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698604.650637:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698604.778484:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698604.778495:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698604.778500:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698604.778510:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.778519:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698604.778527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698604.778563:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146122496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698604.778592:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698604.778595:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698604.778597:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698604.778601:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.778603:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698604.778606:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.778616:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698604.778631:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146122624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698604.778644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698604.778650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787698604.778650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146122496:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698604.778652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698604.778662:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.778666:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698604.778670:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.778675:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698604.778697:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146122752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698604.778708:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698604.778711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698604.778715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698604.778719:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698604.778725:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.778731:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698604.778736:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.778742:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698604.778749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace93480 x1874531146122496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698604.778759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146122880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698604.778771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698604.778774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146122496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698604.778776:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698604.778780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698604.778790:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.778795:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698604.778795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698604.778802:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.778807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698604.778815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698604.778823:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146123008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698604.778823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146122496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698604.778848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146122624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698604.778849:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698604.778856:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698604.778858:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698604.778863:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.778871:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698604.778874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698604.778879:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.778887:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698604.778888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace91500 x1874531146122624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698604.778904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146123136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698604.778908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146122624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698604.778921:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698604.778931:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698604.778936:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698604.778938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698604.778946:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:1.0:1787698604.778946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.778952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146122624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698604.778955:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698604.778958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146122752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698604.778964:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.778971:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698604.778977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698604.778987:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146123264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698604.778989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace91880 x1874531146122752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698604.779021:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698604.779021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146122752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698604.779029:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698604.779034:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698604.779035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698604.779045:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00100000:1.0:1787698604.779047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146122752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698604.779053:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698604.779054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146122880:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698604.779063:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.779071:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698604.779074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698604.779087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace92d80 x1874531146122880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698604.779091:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146123392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698604.779112:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698604.779115:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698604.779119:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698604.779120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146122880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698604.779124:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6504 00000100:00080000:0.0:1787698604.779130:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698604.779137:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698604.779148:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698604.779149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.779160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146122880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698604.779164:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146123520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698604.779176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146123008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698604.779194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698604.779205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace90e00 x1874531146123008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698604.779213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146123008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698604.779219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.779223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146123008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698604.779226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698604.779236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698604.779242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace91180 x1874531146123136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698604.779249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146123136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698604.779255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.779259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698604.779262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698604.779271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698604.779277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace93480 x1874531146123264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698604.779284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146123264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698604.779289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.779292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698604.779295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698604.779304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698604.779310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace90380 x1874531146123392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698604.779317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146123392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698604.779322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.779325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698604.779334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698604.779344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698604.779350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698604/real 1787698604] req@ffff8f51ace91c00 x1874531146123520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698604.779399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146123520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698604.779405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698604.779407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698604.779410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146123520:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698605.674656:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698606.698553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698607.722527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698608.746530:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698609.770499:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698609.898474:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898483:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698609.898487:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898496:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898507:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898514:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698609.898551:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146123648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698609.898579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898584:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898586:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898594:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698609.898602:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146123776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898609:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698609.898612:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898615:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898617:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898619:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698609.898629:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146123904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698609.898635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146123648:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698609.898636:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898642:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698609.898645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898648:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898650:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898652:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898655:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698609.898663:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146124032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898671:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698609.898673:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898675:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898677:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898678:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898681:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698609.898687:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146124160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698609.898698:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898700:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898704:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898708:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698609.898708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698609.898717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146124288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698609.898728:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898730:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898732:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898734:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698609.898743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146124416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898747:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698609.898748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698609.898749:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898752:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898756:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898758:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698609.898762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace92300 x1874531146123648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698609.898763:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698609.898776:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146124544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698609.898782:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698609.898784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146123648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698609.898786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698609.898789:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698609.898796:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6509 00000100:00080000:0.0:1787698609.898799:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698609.898802:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698609.898804:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698609.898804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.898809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698609.898812:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146124672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698609.898816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146123648:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698609.898828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146123776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698609.898846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.898856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace93100 x1874531146123776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.898870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146123776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.898878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.898881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.898884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146123776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698609.898912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146123904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698609.898933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.898945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace90700 x1874531146123904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.898957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146123904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.898962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.898964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.898966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146123904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698609.898970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146124032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698609.898984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.898991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace90000 x1874531146124032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.898998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146124032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.899006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.899008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.899010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146124032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698609.899013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146124160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698609.899022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.899027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace93800 x1874531146124160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.899033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146124160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.899038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.899040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.899041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146124160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698609.899044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698609.899053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.899059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace93b80 x1874531146124288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.899066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146124288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.899070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.899072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.899074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698609.899077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787698609.899087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.899092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace92680 x1874531146124416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.899099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146124416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.899106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.899109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.899112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698609.899118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698609.899136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.899147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace91c00 x1874531146124544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.899159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146124544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.899166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.899169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.899173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698609.899178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698609.899197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698609.899206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff8f51ace90380 x1874531146124672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.899214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146124672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.899219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698609.899220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787698609.899222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146124672:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698610.794507:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698611.818573:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698612.842523:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698613.866535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698614.890529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698615.018467:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018475:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698615.018478:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018487:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018496:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018503:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698615.018539:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146124800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018567:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018570:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698615.018571:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018574:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018578:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018582:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018591:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698615.018603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146124928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698615.018617:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018629:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018633:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018638:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698615.018656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146125056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018671:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018678:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698615.018680:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018683:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018687:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018690:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018696:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698615.018706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146125184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698615.018718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018721:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018723:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018726:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018729:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698615.018737:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146125312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018745:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698615.018749:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018752:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018756:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018759:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018763:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698615.018771:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146125440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018778:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018780:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698615.018782:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018785:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018787:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018790:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018794:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698615.018801:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146125568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018808:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018811:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698615.018813:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018816:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018819:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018822:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698615.018832:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146125696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698615.018842:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698615.018844:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698615.018845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698615.018847:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6514 00000100:00080000:3.0:1787698615.018848:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698615.018850:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698615.018853:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698615.018858:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146125824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698615.018940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146124800:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698615.019028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146124928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698615.019056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146125056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698615.019078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146125184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698615.019100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146125312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698615.019128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698615.019150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698615.019171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698615.019182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125824:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698615.914529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698616.938517:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698617.962553:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698618.090857:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787698618.090916:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698618.090932:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698618.090973:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787698618.090989:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698618.090991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f5187388e00 x1874531146124800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698618.091001:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698618.091024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146124800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698618.091031:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698618.091044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1787698618.091046:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698618.091052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146124800:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787698618.091065:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698618.091069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f5187389c00 x1874531146124928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698618.091079:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698618.091089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146124928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698618.091095:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698618.091106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146124928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698618.091123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f518738aa00 x1874531146125056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146125056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146125056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698618.091168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f5187388380 x1874531146125184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146125184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146125184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698618.091198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f5187388a80 x1874531146125312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146125312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146125312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698618.091227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f518738b800 x1874531146125440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146125440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698618.091257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f518738ad80 x1874531146125568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146125568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698618.091287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f518738a300 x1874531146125696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146125696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698618.091324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698615/real 1787698618] req@ffff8f5187389500 x1874531146125824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698618.091334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146125824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698618.091341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698618.091344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698618.091347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146125824:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698618.987751:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698620.010527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698620.138462:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138470:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698620.138473:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138482:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138491:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138504:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698620.138536:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146125952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138569:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138573:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698620.138575:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138582:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138586:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138591:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138601:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698620.138616:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146126080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138625:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138628:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698620.138630:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138633:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138640:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138644:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698620.138655:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146126208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138664:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138667:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698620.138669:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138672:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138675:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138679:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698620.138693:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146126336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138704:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698620.138706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138709:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138712:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138715:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138719:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698620.138728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146126464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138737:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138740:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698620.138742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138746:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138749:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138752:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138756:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698620.138765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146126592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138774:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138777:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698620.138779:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138782:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138785:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138788:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138793:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698620.138802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146126720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138810:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138819:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698620.138821:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138825:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138828:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138832:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138836:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698620.138847:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146126848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698620.138855:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698620.138858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698620.138860:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698620.138863:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6519 00000100:00080000:3.0:1787698620.138866:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698620.138869:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698620.138873:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698620.138882:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146126976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698620.139189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146125952:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787698620.139264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f518738bb80 x1874531146125952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146125952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146125952:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698620.139393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698620.139417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f518738b100 x1874531146126080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146126080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698620.139463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698620.139483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f5187389180 x1874531146126208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146126208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698620.139532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698620.139552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f518738a680 x1874531146126336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146126336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698620.139596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698620.139614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f5187389f80 x1874531146126464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146126464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146126464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698620.139658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698620.139675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f518738b480 x1874531146126592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146126592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698620.139727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698620.139747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f5187388000 x1874531146126720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146126720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698620.139789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698620.139808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f5187389500 x1874531146126848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146126848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698620.139853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698620.139871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698620.139881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698620/real 1787698620] req@ffff8f518738a300 x1874531146126976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698620.139893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146126976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698620.139902:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698620.139905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698620.139908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146126976:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698621.034588:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698622.058530:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698623.082538:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698624.106566:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698625.130527:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698625.258451:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698625.258464:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258472:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00080000:2.0:1787698625.258482:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258489:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698625.258519:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146127104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698625.258547:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258550:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698625.258552:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258556:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00080000:2.0:1787698625.258561:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258565:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.258574:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698625.258591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146127104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698625.258591:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146127232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698625.258614:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698625.258619:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00080000:2.0:1787698625.258628:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.258637:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698625.258652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146127360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698625.258668:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258671:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698625.258672:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258678:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00080000:2.0:1787698625.258682:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258685:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.258690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698625.258700:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698625.258703:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146127488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698625.258715:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258718:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698625.258720:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258726:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00080000:2.0:1787698625.258729:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258734:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.258740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698625.258758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704e680 x1874531146127104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698625.258759:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146127616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698625.258826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258833:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698625.258835:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258841:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00100000:0.0:1787698625.258846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146127104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698625.258848:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258861:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.258874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698625.258875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.258888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.258894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146127104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698625.258902:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146127744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698625.258914:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258918:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698625.258920:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.258924:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00100000:0.0:1787698625.258927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127232:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698625.258934:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.258943:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.258948:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698625.258958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698625.258965:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146127872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698625.258977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704c000 x1874531146127232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698625.258979:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.258994:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698625.258999:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698625.259008:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00100000:0.0:1787698625.259010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146127232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698625.259015:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698625.259019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127232:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698625.259047:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127360:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698625.259060:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698625.259073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698625.259079:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146128000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698625.259083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704d180 x1874531146127360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698625.259132:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698625.259139:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698625.259141:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698625.259145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146127360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698625.259149:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6524 00000100:00080000:2.0:1787698625.259158:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698625.259170:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698625.259182:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698625.259185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698625.259203:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146128128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698625.259204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698625.259232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704fb80 x1874531146127488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698625.259250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146127488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698625.259255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698625.259262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698625.259272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704df80 x1874531146127616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698625.259283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146127616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698625.259288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146127616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698625.259296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146127744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698625.259306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704f800 x1874531146127744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698625.259317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146127744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698625.259323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146127744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698625.259340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146127872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698625.259349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704ce00 x1874531146127872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698625.259405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146127872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698625.259410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146127872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698625.259418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146128000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698625.259431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704c380 x1874531146128000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698625.259444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146128000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698625.259449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146128000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698625.259456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146128128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698625.259464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698625.259469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698625/real 1787698625] req@ffff8f518704e680 x1874531146128128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698625.259475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146128128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698625.259480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698625.259481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698625.259483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146128128:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698626.154553:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698627.178529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698628.202528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698629.226536:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698630.250463:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698630.378472:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698630.378491:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378503:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378513:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378521:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698630.378560:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146128256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698630.378589:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378593:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698630.378595:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378599:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378603:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378607:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378616:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698630.378630:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146128384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698630.378633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146128256:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698630.378648:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698630.378652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378661:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698630.378675:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146128512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698630.378686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378688:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698630.378690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378692:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378694:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378699:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698630.378702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378702:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698630.378712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146128640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698630.378719:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378722:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698630.378723:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378726:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378729:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378731:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378735:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698630.378742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f518738ad80 x1874531146128256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698630.378744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146128768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698630.378773:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378780:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698630.378783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378789:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378792:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378804:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698630.378810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146128256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698630.378817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146128896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698630.378829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698630.378829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378836:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698630.378838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698630.378840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698630.378843:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378847:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698630.378850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146128256:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698630.378852:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378859:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698630.378867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146129024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698630.378873:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698630.378875:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698630.378876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378879:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378882:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698630.378885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128384:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698630.378890:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378896:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698630.378910:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146129152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698630.378920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698630.378933:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698630.378938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f518738b800 x1874531146128384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698630.378939:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698630.378955:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698630.378962:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6530 00000100:00080000:3.0:1787698630.378968:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698630.378974:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698630.378978:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698630.378982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146128384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698630.378990:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146129280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698630.378993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698630.379016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698630.379041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f5187388a80 x1874531146128512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146128512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698630.379082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698630.379096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f5187388380 x1874531146128640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146128640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698630.379122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698630.379131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f518738aa00 x1874531146128768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146128768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146128768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698630.379171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146128896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698630.379191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f5187389c00 x1874531146128896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146128896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146128896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698630.379223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146129024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698630.379235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f5187388e00 x1874531146129024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146129024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146129024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698630.379269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146129152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698630.379279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f518738ad80 x1874531146129152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146129152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146129152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698630.379304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146129280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698630.379313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698630.379318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698630/real 1787698630] req@ffff8f518738a300 x1874531146129280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698630.379325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146129280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698630.379330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698630.379332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698630.379333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146129280:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698631.274544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698632.298502:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698633.322510:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698634.346564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698635.370550:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698635.498494:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498506:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698635.498510:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498521:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498532:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498541:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698635.498580:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146129408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498612:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698635.498618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498623:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498627:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498632:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498641:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698635.498655:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146129536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787698635.498668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146129408:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698635.498670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498673:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498676:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498680:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498684:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698635.498707:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146129664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498718:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698635.498720:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498723:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498726:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498729:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498734:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698635.498742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698635.498743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146129792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498751:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498754:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698635.498756:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498759:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498762:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498766:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498770:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698635.498779:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146129920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498788:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498791:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698635.498793:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698635.498794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace93480 x1874531146129408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698635.498797:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498800:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498804:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498808:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698635.498811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146129408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698635.498817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146130048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698635.498823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698635.498826:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698635.498827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698635.498829:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698635.498831:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698635.498831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146129408:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698635.498835:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498838:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498841:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498844:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698635.498866:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146130176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498875:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498878:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787698635.498878:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129536:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698635.498880:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498883:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00080000:0.0:1787698635.498887:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498890:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498895:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698635.498901:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698635.498905:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146130304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698635.498914:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698635.498917:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698635.498919:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698635.498924:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6535 00000100:00100000:2.0:1787698635.498924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace91180 x1874531146129536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698635.498927:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698635.498931:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698635.498936:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698635.498938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146129536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698635.498945:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146130432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698635.498946:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.498950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.498954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698635.498960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698635.498992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace90e00 x1874531146129664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146129664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698635.499028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698635.499050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace92d80 x1874531146129792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146129792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698635.499081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698635.499092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace91880 x1874531146129920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146129920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146129920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698635.499118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698635.499128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace91500 x1874531146130048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146130048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698635.499152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698635.499162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace90380 x1874531146130176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146130176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698635.499197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698635.499207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace91c00 x1874531146130304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146130304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698635.499242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698635.499261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698635.499273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698635/real 1787698635] req@ffff8f51ace92680 x1874531146130432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698635.499298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146130432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698635.499307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698635.499310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698635.499314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146130432:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698636.394624:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698637.418488:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698638.442569:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698639.466485:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698640.490522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698640.618481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618489:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698640.618493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.618500:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618510:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698640.618516:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698640.618553:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146130560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698640.618584:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618588:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698640.618590:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.618595:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698640.618603:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698640.618614:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698640.618630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146130688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698640.618639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146130560:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698640.618647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698640.618655:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.618660:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618664:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698640.618669:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698640.618675:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698640.618689:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146130816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698640.618699:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698640.618707:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698640.618708:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.618715:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618719:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698640.618724:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698640.618730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698640.618753:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146130944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698640.618764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704c700 x1874531146130560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698640.618777:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698640.618791:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.618801:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618806:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698640.618813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146130560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698640.618813:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698640.618824:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698640.618833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.618843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698640.618843:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146131072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698640.618852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146130560:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698640.618868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618873:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698640.618875:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.618880:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618884:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698640.618885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146130688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698640.618892:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698640.618900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698640.618915:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146131200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698640.618926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698640.618936:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.618942:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698640.618944:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698640.618947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704ed80 x1874531146130688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698640.618952:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.618959:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698640.618967:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698640.618972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146130688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698640.618975:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698640.618996:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146131328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698640.618998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146130688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698640.619017:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698640.619027:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698640.619030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146130816:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698640.619032:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.619043:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:2.0:1787698640.619047:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698640.619052:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698640.619059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698640.619059:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698640.619075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704dc00 x1874531146130816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698640.619077:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146131456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698640.619106:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698640.619108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146130816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698640.619112:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698640.619116:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698640.619127:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6540 00000100:00080000:0.0:1787698640.619132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698640.619133:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698640.619141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698640.619144:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146130816:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698640.619152:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698640.619156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146130944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698640.619172:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146131584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698640.619181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704ea00 x1874531146130944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698640.619208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146130944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698640.619214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146130944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698640.619223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146131072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698640.619234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704e300 x1874531146131072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698640.619246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146131072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698640.619252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146131072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698640.619267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698640.619278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704d880 x1874531146131200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698640.619292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146131200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698640.619297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698640.619304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698640.619313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704d500 x1874531146131328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698640.619327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146131328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698640.619332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698640.619338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698640.619347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704f480 x1874531146131456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698640.619377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146131456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698640.619382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698640.619389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698640.619401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698640.619408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698640/real 1787698640] req@ffff8f518704c700 x1874531146131584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698640.619415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146131584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698640.619421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698640.619423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698640.619425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146131584:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698641.514511:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698642.538554:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698643.562543:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698644.586561:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698645.610561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698645.738466:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738474:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698645.738478:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738488:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738500:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738509:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698645.738550:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146131712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698645.738596:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738600:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698645.738601:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738605:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738608:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698645.738633:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146131840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698645.738643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146131712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698645.738644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698645.738654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738658:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738661:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738663:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698645.738675:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146131968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698645.738700:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738704:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698645.738706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738709:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738712:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738715:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738720:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698645.738727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698645.738731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146132096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698645.738739:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698645.738744:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738747:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738749:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738752:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698645.738762:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146132224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698645.738768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738770:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698645.738771:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698645.738777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f5187389500 x1874531146131712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698645.738778:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738782:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738784:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698645.738798:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146132352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698645.738804:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698645.738810:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698645.738812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146131712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698645.738815:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738820:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738825:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738830:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698645.738835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698645.738841:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146132480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698645.738845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698645.738854:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698645.738856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146131712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698645.738858:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698645.738862:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738867:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738869:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738872:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738874:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698645.738881:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146132608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698645.738889:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146131840:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698645.738889:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698645.738896:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698645.738899:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698645.738903:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6545 00000100:00080000:3.0:1787698645.738906:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698645.738908:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698645.738911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698645.738919:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146132736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698645.738927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.738943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f5187388000 x1874531146131840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.738957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146131840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.738972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.738975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.738979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146131840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698645.738985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146131968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698645.739001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f518738b480 x1874531146131968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146131968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146131968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698645.739042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146132096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698645.739058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f5187389f80 x1874531146132096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146132096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146132096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698645.739111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146132224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698645.739128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f518738a680 x1874531146132224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146132224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146132224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698645.739177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698645.739194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f5187389180 x1874531146132352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146132352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698645.739235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698645.739253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f518738b100 x1874531146132480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146132480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698645.739305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698645.739323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f518738bb80 x1874531146132608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146132608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698645.739386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698645.739404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698645.739414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698645/real 1787698645] req@ffff8f5187389500 x1874531146132736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698645.739426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146132736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698645.739435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698645.739438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698645.739441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146132736:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698646.634564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698647.658534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698648.682505:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698649.706506:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698650.732105:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698650.858465:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858472:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698650.858475:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858483:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858492:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858497:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698650.858527:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146132864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698650.858547:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858550:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698650.858552:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858556:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858559:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858562:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698650.858570:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698650.858582:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146132992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698650.858588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146132864:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698650.858610:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858613:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698650.858614:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858618:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858620:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858623:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698650.858628:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698650.858637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146133120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698650.858643:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858646:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698650.858647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858650:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858653:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858655:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698650.858659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698650.858666:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146133248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698650.858669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698650.858674:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858677:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698650.858678:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858681:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858684:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858687:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698650.858690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698650.858698:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146133376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698650.858705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787698650.858708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace93b80 x1874531146132864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698650.858710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858714:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858716:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858719:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698650.858722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146132864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698650.858723:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698650.858731:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146133504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698650.858732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.858735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698650.858738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698650.858739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146132864:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698650.858742:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698650.858743:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858746:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00080000:0.0:1787698650.858748:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858751:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698650.858754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698650.858763:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146133632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698650.858766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146132992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698650.858784:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.858786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698650.858788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.858792:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6550 00000800:00000100:2.0:1787698650.858792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698650.858794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858798:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698650.858801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698650.858803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace93800 x1874531146132992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698650.858811:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146133760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698650.858814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146132992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698650.858819:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698650.858821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698650.858822:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698650.858824:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698650.858825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698650.858827:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6550 00000100:00100000:2.0:1787698650.858828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146132992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698650.858830:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698650.858833:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698650.858833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146133120:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698650.858844:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698650.858849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698650.858856:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146133888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698650.858859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace90000 x1874531146133120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.858869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146133120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.858875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.858878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.858881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146133120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698650.858886:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146133248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698650.858904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698650.858913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace90700 x1874531146133248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.858930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146133248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.858935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.858937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.858939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146133248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698650.858942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146133376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698650.858953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698650.858959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace93100 x1874531146133376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.858966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146133376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.858970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.858971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.858973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146133376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698650.858976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698650.858984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698650.858990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace92300 x1874531146133504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.858996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146133504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.859004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.859005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.859007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698650.859018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698650.859026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698650.859033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace93b80 x1874531146133632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.859043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146133632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.859049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.859052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.859055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698650.859060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698650.859077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698650.859087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace92680 x1874531146133760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.859094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146133760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.859098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.859107:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.859110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698650.859113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698650.859125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698650.859131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff8f51ace91c00 x1874531146133888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.859137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146133888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.859142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.859143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698650.859145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146133888:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698651.754479:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698652.778530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698653.802483:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698654.826545:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698655.850527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698655.978472:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978492:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698655.978497:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978532:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:0.0:1787698655.978543:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978550:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698655.978587:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146134016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698655.978616:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698655.978621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978626:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:0.0:1787698655.978629:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978633:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698655.978642:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698655.978656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146134144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698655.978664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146134016:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698655.978670:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978675:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698655.978677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978681:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:0.0:1787698655.978686:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978690:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698655.978695:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698655.978707:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146134272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698655.978723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787698655.978729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698655.978734:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978739:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:0.0:1787698655.978743:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978748:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698655.978752:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698655.978765:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146134400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698655.978774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace90380 x1874531146134016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698655.978777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698655.978790:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978797:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00100000:2.0:1787698655.978801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146134016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698655.978802:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698655.978822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698655.978822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.978830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.978835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146134016:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698655.978838:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146134528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698655.978858:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978862:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698655.978863:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978868:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00100000:2.0:1787698655.978873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134144:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698655.978874:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978881:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698655.978886:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698655.978897:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146134656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698655.978905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698655.978906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978911:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698655.978912:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.978917:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:0.0:1787698655.978920:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698655.978925:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698655.978926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace91500 x1874531146134144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698655.978929:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698655.978940:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146134784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698655.978947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146134144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698655.978951:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.978959:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698655.978964:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698655.978965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698655.978970:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:2.0:1787698655.978973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698655.978976:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698655.978980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134144:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698655.978985:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698655.978991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134272:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698655.978992:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698655.979008:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146134912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698655.979019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698655.979021:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698655.979024:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698655.979026:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698655.979030:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6555 00000100:00080000:0.0:1787698655.979033:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698655.979036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace91880 x1874531146134272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698655.979038:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698655.979044:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698655.979055:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146135040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698655.979060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146134272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698655.979084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698655.979101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698655.979111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace92d80 x1874531146134400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.979122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146134400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698655.979142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698655.979156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698655.979165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace90e00 x1874531146134528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.979176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146134528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146134528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698655.979204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146134656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698655.979220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698655.979230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace91180 x1874531146134656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.979241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146134656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146134656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698655.979261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146134784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698655.979276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698655.979286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace93480 x1874531146134784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.979296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146134784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146134784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698655.979325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146134912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698655.979341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698655.979351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace90380 x1874531146134912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.979387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146134912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146134912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698655.979406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146135040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698655.979423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698655.979435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698655] req@ffff8f51ace91c00 x1874531146135040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.979445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146135040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698655.979453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698655.979456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698655.979459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146135040:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698656.874524:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698657.898531:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698658.922526:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698659.946573:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698660.970504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698661.098490:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.098504:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698661.098509:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098533:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098548:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098557:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698661.098614:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146135168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698661.098645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.098649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698661.098651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098664:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.098675:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698661.098687:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146135296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698661.098697:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698661.098702:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146135168:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698661.098703:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698661.098706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098715:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098718:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098721:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.098725:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698661.098743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146135424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698661.098758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.098761:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698661.098763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098766:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098770:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098774:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.098779:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698661.098792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698661.098792:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146135552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698661.098803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.098807:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698661.098808:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098811:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098816:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098819:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.098823:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698661.098832:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146135680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698661.098841:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.098844:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698661.098846:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098851:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098856:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098860:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.098865:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698661.098875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f518738a300 x1874531146135168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698661.098878:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146135808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698661.098906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698661.098907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146135168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698661.098921:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698661.098923:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698661.098928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698661.098929:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098935:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698661.098938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698661.098944:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698661.098946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146135168:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698661.098953:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698661.098967:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146135936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698661.098977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135296:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698661.098978:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.098983:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698661.098984:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.098989:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00080000:3.0:1787698661.098993:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.098998:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.099003:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698661.099014:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146136064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698661.099024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698661.099034:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698661.099037:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787698661.099041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f518738ad80 x1874531146135296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698661.099047:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698661.099054:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6560 00000100:00100000:2.0:1787698661.099057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146135296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698661.099059:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698661.099067:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698661.099076:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698661.099080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698661.099090:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146136192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698661.099095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698661.099102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698661.099125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f5187388e00 x1874531146135424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146135424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698661.099166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698661.099185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f5187389c00 x1874531146135552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146135552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698661.099225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698661.099241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f518738aa00 x1874531146135680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146135680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146135680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698661.099280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146135808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698661.099295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f5187388380 x1874531146135808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146135808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146135808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698661.099345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146135936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698661.099393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f5187388a80 x1874531146135936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146135936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146135936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698661.099445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146136064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698661.099464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f518738b800 x1874531146136064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146136064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146136064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698661.099504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146136192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698661.099517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698661.099526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698661/real 1787698661] req@ffff8f518738a300 x1874531146136192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698661.099535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146136192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698661.099542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698661.099545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698661.099548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146136192:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698661.994555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698663.018845:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698664.042522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698665.066515:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698666.090613:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698666.218532:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218546:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698666.218551:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218561:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218575:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218581:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698666.218629:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146136320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218664:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698666.218665:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218673:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218676:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218679:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218687:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698666.218699:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146136448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218708:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698666.218711:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218715:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218718:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698666.218724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146136320:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698666.218725:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218733:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698666.218746:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146136576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698666.218760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218765:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218767:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698666.218780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146136704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218786:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218788:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698666.218789:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218791:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218793:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218795:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698666.218806:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146136832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218811:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218816:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787698666.218817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698666.218818:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218824:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218826:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218829:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218832:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698666.218838:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146136960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218844:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218847:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698666.218848:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218850:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218852:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218854:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218857:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698666.218863:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146137088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698666.218868:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218870:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698666.218871:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698666.218877:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00100000:0.0:1787698666.218878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704e680 x1874531146136320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698666.218879:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698666.218885:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698666.218891:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698666.218902:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146137216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698666.218908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146136320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698666.218914:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698666.218923:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698666.218926:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698666.218929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698666.218931:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6565 00000100:00080000:2.0:1787698666.218936:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698666.218938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698666.218944:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698666.218946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146136320:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787698666.218949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698666.218959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698666.218964:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146137344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698666.218984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.218998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704c380 x1874531146136448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146136448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698666.219048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698666.219063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704ce00 x1874531146136576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146136576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698666.219100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698666.219115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704f800 x1874531146136704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146136704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698666.219153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698666.219169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704df80 x1874531146136832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146136832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146136832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698666.219207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146136960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698666.219222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704fb80 x1874531146136960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146136960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146136960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698666.219266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146137088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698666.219280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704d180 x1874531146137088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146137088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146137088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698666.219317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146137216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698666.219345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704c000 x1874531146137216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146137216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146137216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698666.219431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146137344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698666.219449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698666.219460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698666/real 1787698666] req@ffff8f518704c700 x1874531146137344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698666.219471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146137344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698666.219479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698666.219481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698666.219484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146137344:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698667.114602:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698668.138567:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698669.162543:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698670.186567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698671.210540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698671.338545:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338559:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698671.338566:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.338581:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.338596:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338604:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698671.338655:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146137472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698671.338690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338695:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698671.338697:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.338703:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.338708:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338713:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.338726:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698671.338734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146137472:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698671.338743:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146137600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698671.338756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698671.338760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.338763:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.338766:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338769:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.338773:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698671.338783:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146137728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698671.338789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338792:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698671.338793:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.338795:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.338797:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338799:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.338802:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698671.338812:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146137856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698671.338820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338823:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698671.338825:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698671.338831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698671.338832:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.338837:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338841:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.338846:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698671.338857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146137984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698671.338867:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338870:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698671.338872:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.338876:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.338879:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338882:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.338887:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698671.338897:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146138112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698671.338906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace92680 x1874531146137472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698671.338917:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.338922:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698671.338924:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698671.338929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146137472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698671.338932:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:2.0:1787698671.338943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.338948:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.338953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146137472:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698671.338979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137600:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698671.338991:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.338998:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.339004:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698671.339008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698671.339021:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146138240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698671.339028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace93b80 x1874531146137600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698671.339036:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.339042:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698671.339047:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.339052:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00100000:2.0:1787698671.339055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146137600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698671.339057:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.339065:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698671.339074:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698671.339074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698671.339091:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146138368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698671.339101:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698671.339103:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698671.339105:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698671.339108:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6570 00000100:00080000:0.0:1787698671.339111:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698671.339115:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137728:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698671.339121:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698671.339134:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146138496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698671.339152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace92300 x1874531146137728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146137728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698671.339198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698671.339221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace93100 x1874531146137856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146137856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698671.339297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698671.339312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace90700 x1874531146137984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146137984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146137984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698671.339346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698671.339387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace90000 x1874531146138112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146138112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698671.339443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698671.339462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace93800 x1874531146138240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146138240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698671.339499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698671.339515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace92680 x1874531146138368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146138368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698671.339553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698671.339571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698671.339578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698671/real 1787698671] req@ffff8f51ace91c00 x1874531146138496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698671.339585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146138496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698671.339591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698671.339592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698671.339594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146138496:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698672.234570:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698673.258602:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698674.282908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698675.306560:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698676.330541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698676.458526:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458536:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698676.458539:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698676.458550:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458561:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458567:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698676.458609:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146138624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698676.458640:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458643:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698676.458645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698676.458650:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458653:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458658:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458668:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698676.458669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146138624:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698676.458681:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146138752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698676.458690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698676.458695:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698676.458697:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458700:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458703:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458707:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698676.458714:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146138880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698676.458722:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458724:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698676.458726:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698676.458729:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458731:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458735:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698676.458743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698676.458747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146139008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698676.458753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458756:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698676.458758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698676.458761:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458764:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458767:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698676.458780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146139136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698676.458788:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698676.458792:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698676.458794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704f480 x1874531146138624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698676.458796:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458799:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458802:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698676.458812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146138624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698676.458815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146139264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698676.458823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698676.458823:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458827:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698676.458828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698676.458829:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698676.458832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146138624:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698676.458832:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458835:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458837:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458842:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698676.458849:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146139392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698676.458857:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698676.458858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146138752:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698676.458860:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698676.458862:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698676.458866:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458868:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698676.458872:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698676.458876:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698676.458877:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698676.458884:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146139520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698676.458888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704d500 x1874531146138752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698676.458893:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698676.458897:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698676.458898:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698676.458900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146138752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698676.458902:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6576 00000100:00080000:2.0:1787698676.458905:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698676.458908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698676.458908:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698676.458911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698676.458913:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698676.458914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146138752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698676.458920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146138880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698676.458921:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146139648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698676.458936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.458945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704d880 x1874531146138880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.458964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146138880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.458975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.458977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.458979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146138880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698676.458984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146139008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698676.458999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.459006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704e300 x1874531146139008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.459013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146139008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.459018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.459020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.459022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146139008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698676.459025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146139136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698676.459033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.459038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704ea00 x1874531146139136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.459049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146139136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.459054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.459055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.459057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146139136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698676.459059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698676.459068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.459074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704dc00 x1874531146139264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.459084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146139264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.459092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.459095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.459098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698676.459116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698676.459136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.459146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704ed80 x1874531146139392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.459171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146139392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.459183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.459186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.459189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698676.459195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698676.459211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.459218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704f480 x1874531146139520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.459225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146139520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.459230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.459232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.459234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698676.459237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698676.459246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698676.459251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698676/real 1787698676] req@ffff8f518704c700 x1874531146139648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698676.459258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146139648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698676.459262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698676.459263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698676.459266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146139648:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698677.354592:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698678.378533:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698679.402539:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698680.426582:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698681.450578:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698681.578496:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578506:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698681.578510:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578521:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578532:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578539:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698681.578582:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146139776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578619:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698681.578624:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578628:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578631:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578634:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578644:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698681.578654:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146139904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578661:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698681.578665:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578668:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578670:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578672:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578675:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698681.578682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146139776:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698681.578685:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146140032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578695:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698681.578698:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578701:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578702:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578704:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698681.578714:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146140160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578719:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578721:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698681.578728:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578731:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578735:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578737:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698681.578747:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146140288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578752:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578755:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698681.578756:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578758:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578760:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578762:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578764:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698681.578771:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146140416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578776:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578778:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698681.578780:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578782:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578785:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578788:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578791:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698681.578802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146139904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698681.578802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146140544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578814:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578817:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698681.578818:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578822:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578825:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578827:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578830:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698681.578836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146140672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698681.578842:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698681.578844:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698681.578845:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698681.578848:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6581 00000100:00080000:3.0:1787698681.578850:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698681.578852:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698681.578855:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698681.578859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146140032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698681.578865:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146140800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698681.578893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146140160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698681.578932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146140288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698681.578953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698681.578976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698681.579012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698681.579032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140800:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698682.474560:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698683.498580:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698684.522578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698684.650724:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787698684.650780:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698684.650802:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650862:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650876:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650886:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650895:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650904:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650912:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650921:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787698684.650931:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698684.651024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f518738a300 x1874531146140800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146140800/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698684.651098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f518738b100 x1874531146140032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146140032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146140032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698684.651137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f5187389180 x1874531146140160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146140160/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146140160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698684.651173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f518738a680 x1874531146140288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146140288/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146140288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698684.651208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f5187389f80 x1874531146140416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146140416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698684.651245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f518738b480 x1874531146140544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146140544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698684.651281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f5187388000 x1874531146140672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146140672/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146140672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698684.651316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f5187389500 x1874531146139776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146139776/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146139776:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698684.651389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698681/real 1787698684] req@ffff8f518738bb80 x1874531146139904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698684.651403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146139904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698684.651411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698684.651414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698684.651418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146139904:192.168.201.125@tcp:38:kworker.0 00010000:00010000:1.0:1787698685.546582:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698686.570507:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698686.698530:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698542:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698686.698546:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698559:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698571:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698580:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698686.698624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146140928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698661:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698665:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698686.698667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698673:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698677:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698681:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698686.698706:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146141056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698686.698721:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698724:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698728:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698731:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698686.698748:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146141184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698757:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698760:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698686.698762:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698766:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698769:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698773:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698778:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698686.698787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146141312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698799:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698686.698801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698805:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698808:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698686.698828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146141440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698838:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698842:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698686.698844:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698848:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698851:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698855:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698686.698871:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146141568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698881:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698884:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698686.698886:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698889:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698892:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698896:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698900:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698686.698911:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146141696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698919:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698923:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698686.698925:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698928:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698931:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698935:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698940:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698686.698953:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146141824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698686.698962:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698686.698965:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698686.698967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698686.698971:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6586 00000100:00080000:0.0:1787698686.698975:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698686.698992:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698686.698997:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698686.699009:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146141952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698686.699324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146140928:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787698686.699434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace90380 x1874531146140928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146140928/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146140928:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698686.699545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698686.699567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace93480 x1874531146141056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146141056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698686.699610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698686.699627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace91180 x1874531146141184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146141184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698686.699666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698686.699682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace90e00 x1874531146141312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146141312/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699716:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698686.699722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698686.699740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace92d80 x1874531146141440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146141440/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146141440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698686.699781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698686.699797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace91880 x1874531146141568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146141568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698686.699836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698686.699851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace91500 x1874531146141696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146141696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698686.699890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698686.699905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace91c00 x1874531146141824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.699924:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146141824/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.699931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.699934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.699937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698686.699943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698686.699958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698686.699967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698686/real 1787698686] req@ffff8f51ace92680 x1874531146141952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698686.700135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146141952/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698686.700146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698686.700149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698686.700153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146141952:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698687.594601:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698688.618507:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698689.642509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698690.666485:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698691.690538:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698691.818473:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818483:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698691.818486:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818503:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818513:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818521:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698691.818564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146142080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818594:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818602:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698691.818604:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818611:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818614:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818618:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818626:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698691.818637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146142208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698691.818651:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818654:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818656:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818659:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818663:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698691.818672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146142336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818686:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698691.818691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818694:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818697:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818700:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818704:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698691.818714:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146142464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818722:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818726:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698691.818727:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818730:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818733:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818737:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818741:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698691.818749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146142592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818761:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698691.818763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818766:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818769:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818773:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818776:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698691.818784:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146142720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818792:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818795:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698691.818797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818799:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818801:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818804:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698691.818814:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146142848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818821:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818824:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698691.818825:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818828:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818837:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818840:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818844:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698691.818852:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146142976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698691.818865:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698691.818868:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698691.818870:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698691.818873:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6591 00000100:00080000:3.0:1787698691.818875:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698691.818878:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698691.818881:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698691.818888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146143104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698691.818949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146142080:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787698691.819007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f518738b800 x1874531146142080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146142080/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146142080:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698691.819094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698691.819114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f5187388a80 x1874531146142208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146142208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698691.819158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698691.819173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f5187388380 x1874531146142336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146142336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698691.819213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698691.819229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f518738aa00 x1874531146142464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146142464/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698691.819269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698691.819283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f5187389c00 x1874531146142592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146142592/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146142592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698691.819323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146142720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698691.819339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f5187388e00 x1874531146142720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146142720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146142720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698691.819406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146142848:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698691.819423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f518738ad80 x1874531146142848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146142848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146142848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698691.819471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146142976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698691.819486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f518738bb80 x1874531146142976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146142976/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146142976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698691.819526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146143104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698691.819541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698691.819549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698691/real 1787698691] req@ffff8f5187389500 x1874531146143104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698691.819560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146143104/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698691.819568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698691.819571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698691.819575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146143104:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698692.714535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698693.738503:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698694.762479:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698695.786484:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698696.810503:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698696.938479:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938489:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698696.938493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938501:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938512:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698696.938518:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698696.938550:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146143232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698696.938576:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698696.938584:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938588:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938592:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698696.938596:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698696.938617:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146143360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698696.938626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146143232:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698696.938642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698696.938645:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938649:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938652:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698696.938654:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938658:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698696.938667:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146143488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698696.938672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938675:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698696.938676:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938678:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938680:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698696.938682:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938686:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698696.938691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698696.938695:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146143616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698696.938701:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698696.938705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938712:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698696.938714:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938716:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698696.938723:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146143744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698696.938729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938732:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698696.938734:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938739:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938743:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698696.938744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183644e00 x1874531146143232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.938750:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938759:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698696.938771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146143232/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.938774:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146143872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698696.938796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698696.938797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698696.938818:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698696.938821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938825:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938829:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698696.938833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698696.938836:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698696.938842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146143232:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787698696.938843:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698696.938864:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146144000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698696.938870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143360:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698696.938880:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938884:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698696.938886:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938891:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938895:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698696.938900:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787698696.938900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938906:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698696.938919:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146144128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698696.938919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183646d80 x1874531146143360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.938932:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698696.938939:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698696.938941:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698696.938944:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6596 00000100:00080000:1.0:1787698696.938947:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698696.938948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146143360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.938951:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698696.938956:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698696.938966:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146144256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698696.938966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.938974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.938981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698696.938987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698696.939006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183646a00 x1874531146143488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146143488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698696.939049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698696.939079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183647b80 x1874531146143616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146143616/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698696.939119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698696.939134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183644380 x1874531146143744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146143744/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146143744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698696.939175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146143872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698696.939192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183647100 x1874531146143872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146143872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146143872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698696.939243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146144000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698696.939259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183645180 x1874531146144000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146144000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146144000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698696.939296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146144128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698696.939311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183645c00 x1874531146144128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146144128/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146144128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698696.939349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146144256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698696.939389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698696.939399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff8f5183647480 x1874531146144256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.939411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146144256/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.939419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698696.939422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698696.939425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146144256:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698697.834480:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698698.858516:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698699.882533:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698700.906521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698701.930492:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698702.058438:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058448:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698702.058453:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698702.058462:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058478:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058489:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698702.058525:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146144384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698702.058550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058554:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698702.058556:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698702.058560:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058563:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058568:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058577:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698702.058586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146144384:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698702.058592:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146144512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698702.058604:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058608:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698702.058610:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698702.058615:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058618:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058623:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058627:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698702.058637:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146144640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698702.058643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698702.058651:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058655:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698702.058657:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698702.058662:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058665:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058669:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698702.058683:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146144768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698702.058691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058694:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698702.058696:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698702.058701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183644700 x1874531146144384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698702.058703:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058727:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058733:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698702.058740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146144384/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698702.058748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146144896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698702.058754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698702.058763:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698702.058763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698702.058770:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787698702.058771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146144384:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698702.058773:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698702.058788:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00100000:2.0:1787698702.058798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144512:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698702.058799:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058809:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058814:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698702.058827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698702.058829:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146145024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698702.058841:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698702.058847:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698702.058854:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00100000:2.0:1787698702.058854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183644000 x1874531146144512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698702.058870:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698702.058872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146144512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698702.058877:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058884:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698702.058894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698702.058896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146145152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698702.058904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698702.058910:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698702.058917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144512:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698702.058920:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698702.058926:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698702.058929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144640:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698702.058934:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058940:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.058944:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698702.058949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698702.058957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698702.058962:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146145280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698702.058971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183646680 x1874531146144640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698702.058973:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698702.058981:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698702.058986:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698702.058989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146144640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698702.058992:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6601 00000100:00080000:1.0:1787698702.058999:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698702.059005:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698702.059010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698702.059012:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698702.059016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698702.059027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698702.059033:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146145408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698702.059052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698702.059067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183647800 x1874531146144768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698702.059078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146144768/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698702.059085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698702.059089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698702.059098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698702.059116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698702.059127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183647480 x1874531146144896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698702.059137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146144896/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698702.059144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698702.059147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146144896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698702.059165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698702.059181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698702.059191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183645c00 x1874531146145024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698702.059201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146145024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698702.059208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698702.059210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698702.059218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698702.059232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698702.059240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183645180 x1874531146145152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698702.059250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146145152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698702.059257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698702.059259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698702.059268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698702.059282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698702.059292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183647100 x1874531146145280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698702.059301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146145280/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698702.059308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698702.059310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698702.059318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698702.059335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698702.059345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698702/real 1787698702] req@ffff8f5183644380 x1874531146145408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698702.059390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146145408/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698702.059399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698702.059402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698702.059406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146145408:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698702.954504:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698703.978533:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698705.002514:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698706.026585:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698707.050574:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698707.178479:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178489:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698707.178493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178503:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178514:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178520:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698707.178561:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146145536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698707.178590:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178593:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698707.178594:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178598:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178600:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178603:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178611:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698707.178619:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146145664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698707.178626:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178639:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698707.178641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178644:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178647:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178649:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178653:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698707.178659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146145536:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698707.178663:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146145792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698707.178674:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178676:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698707.178678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178681:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178683:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178685:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178688:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698707.178695:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146145920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698707.178701:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698707.178704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178711:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178713:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698707.178719:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146146048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698707.178724:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178726:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698707.178728:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178730:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178732:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178734:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178736:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698707.178742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698707.178745:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146146176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698707.178753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178756:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698707.178758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178762:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178765:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178769:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178774:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698707.178786:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146146304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698707.178795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704c000 x1874531146145536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698707.178795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698707.178809:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178816:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00100000:0.0:1787698707.178820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146145536/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698707.178822:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698707.178830:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698707.178833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698707.178835:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698707.178839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.178844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146145536:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698707.178850:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146146432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698707.178851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146145664:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698707.178864:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698707.178866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698707.178867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698707.178870:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6606 00000100:00080000:2.0:1787698707.178874:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698707.178878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698707.178880:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698707.178885:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698707.178891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704d180 x1874531146145664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698707.178896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146146560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698707.178911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146145664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.178918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.178921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.178923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146145664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698707.178944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146145792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698707.178955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.178960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704fb80 x1874531146145792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.178967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146145792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.178972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.178973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.178975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146145792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698707.178978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146145920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698707.178987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.178993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704df80 x1874531146145920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.179003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146145920/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.179008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.179009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.179011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146145920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698707.179014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146146048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698707.179022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.179027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704f800 x1874531146146048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.179033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146146048/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.179038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.179040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.179042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146146048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698707.179044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698707.179053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.179058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704ce00 x1874531146146176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.179064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146146176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.179074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.179075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.179077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698707.179080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698707.179089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.179100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704c380 x1874531146146304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.179112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146146304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.179119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.179122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.179125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698707.179130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698707.179149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.179160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704e680 x1874531146146432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.179167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146146432/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.179171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.179173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.179176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698707.179179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698707.179192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698707.179197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698707/real 1787698707] req@ffff8f518704c700 x1874531146146560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698707.179204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146146560/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698707.179208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698707.179210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698707.179211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146146560:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698708.074548:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698709.098587:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698710.122520:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698711.146559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698712.170524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698712.298450:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698712.298456:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698712.298460:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298480:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298486:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698712.298518:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146146688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698712.298539:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698712.298560:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698712.298562:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298565:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298572:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698712.298579:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698712.298591:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146146816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698712.298592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146146688:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698712.298605:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698712.298611:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698712.298612:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298616:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298619:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298623:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698712.298628:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698712.298640:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146146944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698712.298650:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698712.298651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698712.298652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698712.298655:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298659:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298661:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298665:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698712.298670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698712.298682:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146147072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698712.298691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704f480 x1874531146146688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698712.298692:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698712.298700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698712.298703:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298709:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00100000:0.0:1787698712.298712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146146688/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698712.298713:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298722:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698712.298723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.298730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698712.298732:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698712.298733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146146688:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698712.298749:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146147200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698712.298760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146146816:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698712.298761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698712.298767:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698712.298768:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298771:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298774:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298776:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698712.298780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698712.298789:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146147328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698712.298795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698712.298797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698712.298799:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698712.298801:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298804:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298807:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298809:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698712.298812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704ed80 x1874531146146816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698712.298813:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698712.298824:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146147456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698712.298826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146146816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698712.298842:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698712.298844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698712.298846:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698712.298848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698712.298849:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698712.298854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146146816:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698712.298855:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298860:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698712.298864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146146944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698712.298865:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698712.298871:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698712.298879:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146147584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698712.298885:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698712.298889:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698712.298891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698712.298891:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698712.298897:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6611 00000100:00080000:2.0:1787698712.298900:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698712.298902:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698712.298905:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698712.298909:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704dc00 x1874531146146944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698712.298913:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146147712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698712.298939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146146944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.298949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.298952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.298956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146146944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698712.298961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146147072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698712.298981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698712.298989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704ea00 x1874531146147072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698712.299000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146147072/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.299006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.299009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.299012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146147072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698712.299027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146147200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698712.299044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698712.299059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704e300 x1874531146147200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698712.299066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146147200/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.299071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.299073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.299075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146147200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698712.299080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698712.299097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698712.299107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704d880 x1874531146147328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698712.299118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146147328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.299132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.299139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.299141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698712.299145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698712.299158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698712.299172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704d500 x1874531146147456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698712.299179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146147456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.299183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.299185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.299187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698712.299190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698712.299200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698712.299207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704f480 x1874531146147584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698712.299216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146147584/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.299223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.299226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.299229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698712.299235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698712.299251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698712.299265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698712/real 1787698712] req@ffff8f518704c700 x1874531146147712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698712.299272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146147712/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698712.299277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698712.299278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698712.299280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146147712:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698713.194499:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698714.218527:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698715.242525:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698716.266557:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698717.290473:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698717.418490:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418498:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698717.418501:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418508:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418517:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418522:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698717.418551:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146147840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418572:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418575:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698717.418576:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418579:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418581:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418583:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418590:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698717.418598:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146147968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418607:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698717.418612:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418619:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00100000:2.0:1787698717.418620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146147840:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698717.418623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418630:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418635:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698717.418648:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146148096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418655:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698717.418658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418661:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418664:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418666:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698717.418677:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146148224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418682:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418686:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698717.418688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698717.418688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698717.418694:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418696:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418698:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418701:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698717.418708:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146148352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698717.418716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418718:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418721:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418723:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418725:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698717.418732:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146148480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698717.418738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace93800 x1874531146147840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698717.418740:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418746:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698717.418748:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418756:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418760:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418762:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418765:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698717.418770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146147840/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698717.418772:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146148608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698717.418801:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698717.418803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698717.418803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698717.418810:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418813:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698717.418814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698717.418818:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698717.418823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146147840:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698717.418828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146148736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698717.418836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698717.418836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146147968:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698717.418845:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698717.418846:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698717.418851:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6617 00000100:00080000:0.0:1787698717.418854:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698717.418858:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698717.418863:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698717.418868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698717.418877:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146148864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698717.418886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace90000 x1874531146147968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.418913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146147968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.418922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.418926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.418930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146147968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698717.418953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146148096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698717.418973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.418985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace90700 x1874531146148096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.418997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146148096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146148096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698717.419017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146148224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698717.419036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.419048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace93100 x1874531146148224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.419059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146148224/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146148224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698717.419078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146148352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698717.419096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.419108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace92300 x1874531146148352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.419120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146148352/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146148352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698717.419139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698717.419157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.419168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace93b80 x1874531146148480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.419187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146148480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698717.419207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698717.419225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.419236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace92680 x1874531146148608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.419248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146148608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698717.419267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698717.419285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.419297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace91c00 x1874531146148736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.419308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146148736/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698717.419333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698717.419350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698717.419382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698717/real 1787698717] req@ffff8f51ace91500 x1874531146148864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698717.419394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146148864/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698717.419402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698717.419404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698717.419407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146148864:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698718.314578:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698719.338623:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698720.362669:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698721.386592:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698722.410589:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698722.538504:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538515:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698722.538519:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538529:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538541:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538548:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698722.538588:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146148992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538613:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538616:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698722.538617:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538620:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538623:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538633:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698722.538643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146149120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538650:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538652:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698722.538654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538656:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538659:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538662:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698722.538665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146148992:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698722.538666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698722.538675:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146149248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538681:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538683:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698722.538684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538686:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538689:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538691:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538694:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698722.538700:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146149376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538707:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698722.538709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538711:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538714:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538715:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538718:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698722.538725:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146149504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538738:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698722.538739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538741:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538743:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538745:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698722.538750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698722.538760:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146149632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538766:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538769:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698722.538770:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538772:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538776:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538777:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538780:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698722.538788:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146149760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538793:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698722.538796:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698722.538797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538800:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00100000:2.0:1787698722.538803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace91880 x1874531146148992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698722.538804:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538808:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538811:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698722.538819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146149888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698722.538826:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698722.538829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146148992/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698722.538830:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698722.538833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698722.538840:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6622 00000100:00080000:0.0:1787698722.538844:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698722.538849:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698722.538852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698722.538855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698722.538860:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146150016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698722.538864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.538873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146148992:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698722.538882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698722.538907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.538920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace92d80 x1874531146149120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.538931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146149120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.538939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.538942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.538946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698722.538974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698722.538992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace90e00 x1874531146149248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146149248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698722.539033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698722.539050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace91180 x1874531146149376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146149376/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698722.539091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698722.539108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace93480 x1874531146149504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146149504/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146149504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698722.539148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146149632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698722.539165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace90380 x1874531146149632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146149632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146149632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698722.539210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146149760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698722.539225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace91500 x1874531146149760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146149760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146149760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698722.539265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146149888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698722.539285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace91c00 x1874531146149888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146149888/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146149888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698722.539326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146150016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698722.539346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698722.539398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698722/real 1787698722] req@ffff8f51ace92680 x1874531146150016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698722.539410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146150016/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698722.539419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698722.539422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698722.539424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146150016:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698723.434559:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698724.458510:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698725.482527:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698726.506525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698727.530529:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698727.658444:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658455:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698727.658458:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658468:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658479:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698727.658486:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698727.658520:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146150144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698727.658546:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658549:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698727.658550:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658554:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658558:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698727.658562:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658571:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698727.658586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146150144:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698727.658587:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146150272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698727.658602:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658605:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698727.658607:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658611:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658614:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698727.658617:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658623:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698727.658633:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146150400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698727.658641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698727.658646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698727.658651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698727.658652:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658657:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698727.658670:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658675:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698727.658687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146150528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698727.658692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183647b80 x1874531146150144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698727.658702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698727.658709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698727.658711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146150144/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698727.658716:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658722:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698727.658725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698727.658729:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698727.658731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.658737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146150144:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787698727.658738:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698727.658756:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146150656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698727.658765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150272:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698727.658768:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658774:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698727.658776:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658781:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658784:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698727.658787:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658793:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698727.658797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698727.658810:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146150784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698727.658815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183646a00 x1874531146150272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698727.658830:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658836:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698727.658838:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658842:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00080000:1.0:1787698727.658846:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698727.658848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146150272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698727.658850:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658856:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698727.658866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698727.658869:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146150912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698727.658873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698727.658882:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698727.658883:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150272:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698727.658886:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698727.658889:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658895:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6627 00000100:00100000:2.0:1787698727.658895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150400:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698727.658899:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698727.658904:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658907:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698727.658915:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146151040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698727.658920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698727.658922:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698727.658924:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698727.658929:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6627 00000800:00000100:2.0:1787698727.658934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698727.658936:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698727.658948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183646d80 x1874531146150400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698727.658955:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698727.658964:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698727.658968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146150400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.658977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.658980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.658984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698727.658987:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146151168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698727.658999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698727.659023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698727.659033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183644e00 x1874531146150528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698727.659044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146150528/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.659051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.659054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.659057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698727.659073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698727.659092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698727.659103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183644380 x1874531146150656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698727.659113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146150656/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.659119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.659122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.659125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146150656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698727.659130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146150784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698727.659146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698727.659155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183647100 x1874531146150784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698727.659165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146150784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.659173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.659176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.659179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146150784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698727.659185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146150912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698727.659198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698727.659205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183645180 x1874531146150912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698727.659212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146150912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.659216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.659218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.659220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146150912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698727.659223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146151040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698727.659231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698727.659236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183645c00 x1874531146151040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698727.659242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146151040/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.659246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.659247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.659249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146151040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698727.659252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146151168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698727.659260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698727.659265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698727/real 1787698727] req@ffff8f5183647480 x1874531146151168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698727.659271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146151168/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698727.659275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698727.659277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698727.659301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146151168:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698728.554579:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698729.578547:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698730.602512:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698731.626540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698732.650522:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698732.778459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778466:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698732.778469:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778476:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778483:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778489:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698732.778518:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146151296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698732.778541:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778544:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698732.778545:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778547:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778549:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778552:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698732.778558:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698732.778564:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146151424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698732.778570:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778573:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698732.778573:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778576:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778578:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778580:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698732.778582:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698732.778588:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146151552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698732.778593:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778596:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698732.778598:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778601:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778603:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778605:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146151296:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698732.778608:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698732.778614:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146151680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698732.778620:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698732.778623:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778625:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778628:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778629:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698732.778632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698732.778639:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146151808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698732.778644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778646:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698732.778647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778649:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778651:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778653:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698732.778656:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698732.778665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698732.778666:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146151936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698732.778676:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778680:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698732.778682:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778685:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00080000:0.0:1787698732.778689:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778693:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698732.778699:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698732.778700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace93b80 x1874531146151296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698732.778713:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146152064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698732.778716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146151296/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698732.778732:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698732.778735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698732.778736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698732.778739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698732.778742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698732.778745:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00100000:2.0:1787698732.778749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146151296:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698732.778750:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778756:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151424:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698732.778762:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698732.778774:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146152192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698732.778781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698732.778782:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698732.778785:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698732.778787:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698732.778791:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6632 00000100:00100000:2.0:1787698732.778791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace92300 x1874531146151424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698732.778796:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698732.778801:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146151424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698732.778810:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698732.778822:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146152320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698732.778822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.778829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.778835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698732.778854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698732.778866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace93100 x1874531146151552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.778879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146151552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.778883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.778891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.778893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698732.778896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698732.778907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace90700 x1874531146151680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.778919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146151680/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.778923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.778925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.778927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698732.778930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698732.778939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace90000 x1874531146151808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.778950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146151808/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.778954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.778955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.778957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146151808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698732.778960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146151936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698732.778968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.778973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace93800 x1874531146151936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.778979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146151936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.778984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.778985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.778987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146151936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698732.778989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146152064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698732.778997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.779003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace92680 x1874531146152064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.779009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146152064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.779013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.779015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.779016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146152064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698732.779019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146152192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698732.779026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.779031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace91c00 x1874531146152192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.779040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146152192/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.779047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.779050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.779053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146152192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698732.779058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146152320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698732.779074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698732.779083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698732/real 1787698732] req@ffff8f51ace91500 x1874531146152320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698732.779093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146152320/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698732.779100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698732.779102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698732.779105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146152320:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698733.674548:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698734.698550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698735.722591:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698736.746517:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698737.770502:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698737.898471:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898481:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698737.898486:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898497:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898508:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898514:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698737.898561:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146152448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698737.898591:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898594:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698737.898595:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898600:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898603:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898606:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898615:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698737.898626:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146152576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698737.898633:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898636:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698737.898637:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898640:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898642:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898646:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698737.898649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146152448:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698737.898649:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698737.898660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146152704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698737.898666:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898668:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698737.898669:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898674:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898676:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698737.898686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146152832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698737.898691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698737.898694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898697:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898709:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898712:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898715:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698737.898722:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146152960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698737.898727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698737.898733:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698737.898737:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898740:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898743:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898745:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698737.898755:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146153088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698737.898761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698737.898765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898767:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898770:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898773:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898777:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698737.898787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f5187388000 x1874531146152448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698737.898787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146153216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698737.898810:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898813:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698737.898814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698737.898817:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898819:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698737.898823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146152448/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698737.898825:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898830:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698737.898839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698737.898841:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146153344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698737.898845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.898852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146152448:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698737.898855:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698737.898859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698737.898860:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698737.898862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152576:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698737.898866:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6637 00000100:00080000:3.0:1787698737.898870:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698737.898872:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698737.898875:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698737.898883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146153472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698737.898890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.898905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f518738b480 x1874531146152576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.898917:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146152576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.898926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.898929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.898932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698737.898961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698737.898981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.898992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f5187389f80 x1874531146152704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146152704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698737.899018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698737.899032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.899039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f518738a680 x1874531146152832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146152832/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698737.899063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698737.899080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.899091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f5187389180 x1874531146152960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146152960/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146152960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698737.899127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698737.899142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.899149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f518738b100 x1874531146153088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146153088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698737.899168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698737.899177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.899185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f518738a300 x1874531146153216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146153216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698737.899221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698737.899236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.899243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f5187389500 x1874531146153344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146153344/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698737.899261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698737.899270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698737.899275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff8f518738bb80 x1874531146153472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.899281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146153472/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.899285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698737.899287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698737.899289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146153472:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698738.794518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698739.818493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698740.842518:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698741.866541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698742.890605:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698743.018471:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018481:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698743.018485:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018496:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018507:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018514:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698743.018554:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146153600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018579:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698743.018583:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018587:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018589:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018591:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018600:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698743.018608:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146153728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698743.018619:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018623:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00100000:2.0:1787698743.018625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146153600:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698743.018625:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018629:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018635:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698743.018643:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146153856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018653:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018655:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698743.018656:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018659:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018661:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018663:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018666:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698743.018673:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146153984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018678:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018681:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698743.018682:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018684:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018686:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018687:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018690:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698743.018696:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146154112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018701:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698743.018706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698743.018707:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698743.018709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018714:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018717:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698743.018731:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146154240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018735:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018737:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698743.018738:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698743.018740:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018743:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018746:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698743.018754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146154368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698743.018759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018761:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698743.018764:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698743.018765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f518738ad80 x1874531146153600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698743.018768:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018773:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698743.018778:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018783:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698743.018795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146153600/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698743.018795:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146154496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698743.018825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698743.018826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698743.018831:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698743.018833:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698743.018834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698743.018837:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6642 00000100:00080000:3.0:1787698743.018840:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698743.018841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146153600:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698743.018845:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698743.018850:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698743.018855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146154624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698743.018867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146153728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698743.018886:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.018897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f5187388e00 x1874531146153728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.018904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146153728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.018910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.018912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.018914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146153728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698743.018917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146153856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698743.018926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.018933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f5187389c00 x1874531146153856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.018939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146153856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.018943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.018945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.018947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146153856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698743.018949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146153984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698743.018961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.018971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f518738aa00 x1874531146153984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.018981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146153984/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.018988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.018991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.018994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146153984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698743.019000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146154112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698743.019033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.019045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f5187388380 x1874531146154112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.019055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146154112/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.019061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.019063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.019065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146154112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698743.019068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698743.019082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.019089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f5187388a80 x1874531146154240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.019095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146154240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.019101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.019102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.019104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698743.019107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154368:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698743.019116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.019122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f518738b800 x1874531146154368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.019128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146154368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.019132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.019134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.019136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698743.019139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698743.019148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.019153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f518738bb80 x1874531146154496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.019158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146154496/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.019162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.019164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.019166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698743.019169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698743.019178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698743.019182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698743/real 1787698743] req@ffff8f5187389500 x1874531146154624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698743.019188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146154624/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698743.019192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698743.019194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698743.019195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146154624:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698743.914512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698744.938546:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698745.962607:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698746.986519:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698748.010613:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698748.138495:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138506:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698748.138511:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138522:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138535:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138543:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698748.138586:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146154752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698748.138615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698748.138620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138627:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138631:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138641:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698748.138653:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146154880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698748.138656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146154752:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698748.138663:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698748.138668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138671:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138673:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138675:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138678:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698748.138686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146155008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698748.138697:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698748.138702:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138705:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138707:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138710:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138713:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698748.138720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146155136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698748.138725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698748.138729:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138732:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138734:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138737:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698748.138748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146155264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698748.138753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138756:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698748.138758:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138761:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138769:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138773:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698748.138782:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146155392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698748.138792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146154880:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698748.138793:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138802:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698748.138804:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138810:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138816:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138821:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138827:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698748.138843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146155008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698748.138844:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146155520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698748.138860:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698748.138863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698748.138865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138869:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138873:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138876:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138880:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698748.138881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146155136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698748.138895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146155648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698748.138907:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698748.138910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146155264:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698748.138913:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698748.138917:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698748.138921:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6647 00000100:00080000:2.0:1787698748.138924:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698748.138927:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698748.138943:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698748.138950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698748.138959:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146155776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698748.138977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698748.138992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698748.139007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155776:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698749.034547:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698750.058555:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698751.082565:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698751.210706:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787698751.210756:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698751.210774:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698751.210835:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787698751.210860:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698751.210874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704c700 x1874531146155776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698751.210875:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698751.210902:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698751.210922:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146155776/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698751.210925:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698751.210941:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698751.210945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1787698751.210953:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:0.0:1787698751.210955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.210964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787698751.210970:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787698751.210980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704fb80 x1874531146155392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.210998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146155392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698751.211027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704d180 x1874531146155520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146155520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698751.211069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704c000 x1874531146155648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146155648/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146155648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698751.211128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704c380 x1874531146154880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146154880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146154880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698751.211176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704ce00 x1874531146155008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146155008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146155008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698751.211216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704f800 x1874531146155136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146155136/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146155136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698751.211255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704df80 x1874531146155264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146155264/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146155264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698751.211299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698748/real 1787698751] req@ffff8f518704e680 x1874531146154752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698751.211314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146154752/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698751.211324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698751.211328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698751.211333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146154752:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787698752.106487:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698753.130556:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698753.258477:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.258485:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698753.258488:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.258496:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:0.0:1787698753.258505:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698753.258520:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698753.258557:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146155904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698753.258591:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.258595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698753.258597:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.258610:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:0.0:1787698753.258614:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698753.258618:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698753.258628:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698753.258644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146156032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698753.258649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146155904:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698753.258661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.258672:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698753.258674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.258680:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:0.0:1787698753.258691:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698753.258696:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698753.258702:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698753.258721:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146156160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698753.258736:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.258739:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698753.258743:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698753.258745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698753.258750:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:0.0:1787698753.258756:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698753.258760:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698753.258766:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698753.258778:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146156288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698753.258789:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.258792:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698753.258794:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.258800:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00100000:2.0:1787698753.258805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace90380 x1874531146155904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698753.258807:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698753.258816:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698753.258825:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698753.258839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146155904/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698753.258857:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146156416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698753.258875:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.258883:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698753.258888:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.258898:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:2.0:1787698753.258899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698753.258904:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698753.258909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698753.258914:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698753.258920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146155904:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698753.258923:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698753.258942:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146156544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698753.258954:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698753.258956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156032:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698753.258959:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698753.258964:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.258972:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:0.0:1787698753.258976:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698753.258982:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787698753.259000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698753.259001:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698753.259019:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146156672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698753.259021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace93480 x1874531146156032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146156032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698753.259056:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698753.259065:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698753.259069:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698753.259072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698753.259077:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:2.0:1787698753.259083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698753.259086:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698753.259091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156032:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698753.259095:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698753.259104:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698753.259106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698753.259120:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146156800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698753.259132:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698753.259138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698753.259139:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698753.259144:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698753.259149:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6652 00000100:00080000:0.0:1787698753.259154:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698753.259159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace91180 x1874531146156160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698753.259162:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698753.259171:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698753.259187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146156160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698753.259191:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146156928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698753.259218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698753.259240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698753.259265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698753.259279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace90e00 x1874531146156288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146156288/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698753.259316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698753.259333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698753.259388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698753.259404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace92d80 x1874531146156416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146156416/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698753.259427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146156416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698753.259456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698753.259479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698753.259492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace91880 x1874531146156544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146156544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698753.259515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698753.259536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698753.259556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698753.259569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace90380 x1874531146156672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146156672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698753.259587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698753.259599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698753.259617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698753.259629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace91500 x1874531146156800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146156800/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698753.259646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698753.259657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698753.259674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698753.259686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698753/real 1787698753] req@ffff8f51ace91c00 x1874531146156928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698753.259698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146156928/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698753.259706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698753.259709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698753.259712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146156928:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698754.154534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698755.178517:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698756.202475:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698757.226540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698758.250558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698758.378484:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378493:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698758.378497:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378506:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378516:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378521:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698758.378552:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146157056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698758.378575:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378577:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698758.378579:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378582:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378585:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378588:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378596:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698758.378610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146157056:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698758.378610:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146157184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698758.378641:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698758.378648:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378653:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378657:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378661:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378667:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698758.378672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698758.378686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146157312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698758.378698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698758.378704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698758.378710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704f480 x1874531146157056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698758.378711:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378717:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378723:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698758.378737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146157056/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698758.378742:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146157440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698758.378771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698758.378775:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698758.378781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698758.378782:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378790:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00100000:0.0:1787698758.378791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146157056:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698758.378794:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378801:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378805:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698758.378815:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146157568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698758.378821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157184:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698758.378827:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378831:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698758.378832:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378835:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378837:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378840:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698758.378855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146157696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698758.378856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698758.378865:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378869:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698758.378870:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378873:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378875:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698758.378879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704d500 x1874531146157184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698758.378880:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378885:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698758.378895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146157824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698758.378907:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698758.378908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146157184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698758.378912:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698758.378915:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378922:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:0.0:1787698758.378926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698758.378926:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698758.378932:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698758.378935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698758.378937:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698758.378940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698758.378949:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146157952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698758.378950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157312:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698758.378963:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698758.378966:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698758.378968:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698758.378971:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6658 00000100:00080000:2.0:1787698758.378974:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698758.378978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698758.378979:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698758.378984:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698758.378992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704d880 x1874531146157312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698758.378996:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146158080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698758.379015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146157312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698758.379032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698758.379045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698758.379052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704e300 x1874531146157440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698758.379059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146157440/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698758.379079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698758.379089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698758.379094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704ea00 x1874531146157568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698758.379102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146157568/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146157568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698758.379114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146157696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698758.379132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698758.379142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704dc00 x1874531146157696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698758.379154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146157696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146157696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698758.379166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146157824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698758.379176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698758.379182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704ed80 x1874531146157824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698758.379189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146157824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146157824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698758.379201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146157952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698758.379210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698758.379215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704f480 x1874531146157952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698758.379222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146157952/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146157952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698758.379239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146158080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698758.379248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698758.379253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698758/real 1787698758] req@ffff8f518704e680 x1874531146158080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698758.379259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146158080/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698758.379264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698758.379266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698758.379268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146158080:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698759.274530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698760.298526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698761.322540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698762.346519:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698763.370589:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698763.498478:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698763.498489:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698763.498493:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698763.498504:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.498515:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698763.498524:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698763.498563:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146158208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698763.498598:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698763.498602:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698763.498604:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698763.498610:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.498614:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698763.498619:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698763.498630:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698763.498631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146158208:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698763.498648:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146158336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698763.498661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698763.498665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698763.498667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698763.498686:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.498690:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698763.498695:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698763.498701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698763.498711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698763.498716:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146158464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698763.498729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698763.498732:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698763.498745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698763.498756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace92680 x1874531146158208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.498757:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.498764:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698763.498772:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698763.498782:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698763.498792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146158208/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698763.498799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146158592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698763.498811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698763.498815:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698763.498820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698763.498822:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698763.498827:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698763.498829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146158208:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698763.498835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.498840:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698763.498844:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698763.498852:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698763.498866:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146158720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698763.498871:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158336:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698763.498888:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698763.498893:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698763.498895:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698763.498903:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6663 00000800:00000100:2.0:1787698763.498904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698763.498911:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698763.498919:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698763.498920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace93800 x1874531146158336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698763.498928:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698763.498933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146158336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698763.498940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698763.498945:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146158848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698763.498946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.498953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698763.498965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158464:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698763.498983:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698763.498988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698763.498992:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698763.498996:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698763.499000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace90000 x1874531146158464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.499004:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.499013:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698763.499019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146158464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.499023:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698763.499032:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698763.499043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698763.499046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698763.499055:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146158976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698763.499059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158592:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698763.499076:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698763.499082:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787698763.499082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698763.499087:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698763.499091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace90700 x1874531146158592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.499093:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.499101:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698763.499106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146158592/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.499109:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698763.499120:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698763.499120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698763.499125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698763.499153:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146159104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698763.499154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158720:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698763.499173:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698763.499174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698763.499181:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698763.499184:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698763.499186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace93100 x1874531146158720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.499192:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6663 00000100:00080000:0.0:1787698763.499201:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698763.499203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146158720/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698763.499209:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698763.499215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698763.499218:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698763.499220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146158720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698763.499232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146158848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698763.499241:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146159232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698763.499248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698763.499260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace92300 x1874531146158848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698763.499269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146158848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698763.499275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698763.499277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146158848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698763.499284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146158976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698763.499294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698763.499300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace93b80 x1874531146158976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698763.499308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146158976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698763.499312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698763.499314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146158976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698763.499324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146159104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698763.499333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698763.499340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace92680 x1874531146159104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698763.499347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146159104/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698763.499352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698763.499370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146159104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698763.499376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146159232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698763.499388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698763.499395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698763/real 1787698763] req@ffff8f51ace91c00 x1874531146159232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698763.499403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146159232/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698763.499408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698763.499410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698763.499412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146159232:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698764.394507:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698765.418529:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698766.442461:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698767.466529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698768.490632:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698768.618464:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618480:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698768.618485:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618506:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618519:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.618527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698768.618570:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146159360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.618614:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618619:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698768.618624:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618630:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618635:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.618641:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698768.618655:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698768.618665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146159360:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698768.618674:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146159488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.618688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698768.618694:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618698:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618702:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.618707:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698768.618714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698768.618729:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146159616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.618741:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618745:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698768.618747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618751:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618755:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698768.618761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698768.618762:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698768.618770:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698768.618785:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146159744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.618796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618799:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698768.618801:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618805:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618808:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.618812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698768.618818:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698768.618830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f518738a300 x1874531146159360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698768.618835:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146159872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.618860:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618864:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698768.618866:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698768.618869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146159360/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698768.618874:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618880:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.618887:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698768.618891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698768.618893:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698768.618901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698768.618911:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146160000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698768.618924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146159360:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698768.618929:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618935:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698768.618937:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618943:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:3.0:1787698768.618948:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.618952:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698768.618957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698768.618961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698768.618974:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146160128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.618984:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.618986:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698768.618988:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.618992:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6668 00000800:00000100:2.0:1787698768.618997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698768.618997:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698768.619004:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698768.619009:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698768.619019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f518738b100 x1874531146159488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698768.619025:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146160256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698768.619051:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698768.619056:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787698768.619057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146159488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698768.619060:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698768.619071:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6668 00000100:00080000:2.0:1787698768.619073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698768.619075:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698768.619081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698768.619083:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159488:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698768.619090:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698768.619096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698768.619116:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146160384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698768.619121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f5187389180 x1874531146159616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146159616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698768.619192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698768.619216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f518738a680 x1874531146159744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146159744/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698768.619278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698768.619296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f5187389f80 x1874531146159872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146159872/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146159872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698768.619337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698768.619393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f518738b480 x1874531146160000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146160000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698768.619463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698768.619496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f5187388000 x1874531146160128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146160128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698768.619542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698768.619563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f518738a300 x1874531146160256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146160256/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698768.619627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698768.619649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698768.619658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698768/real 1787698768] req@ffff8f5187389500 x1874531146160384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698768.619666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146160384/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698768.619672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698768.619674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698768.619676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146160384:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698769.514626:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698770.538599:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698771.562580:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698772.586552:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698773.610589:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698773.738577:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.738591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698773.738596:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.738609:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.738625:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.738633:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698773.738684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146160512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698773.738724:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.738729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698773.738731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.738745:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.738753:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.738759:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698773.738771:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698773.738787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146160640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698773.738792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146160512:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698773.738805:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.738809:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698773.738811:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.738816:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.738820:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.738824:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698773.738829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698773.738842:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146160768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698773.738852:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.738856:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698773.738859:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.738863:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.738868:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.738872:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698773.738878:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698773.738891:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146160896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698773.738902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698773.738909:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.738913:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698773.738916:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.738922:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.738927:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.738931:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698773.738937:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698773.738948:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146161024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698773.738958:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.738962:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698773.738964:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.738969:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.738972:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.738977:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698773.738983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704df80 x1874531146160512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698773.738985:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698773.739002:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146161152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698773.739013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146160512/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698773.739015:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.739025:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698773.739030:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698773.739036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698773.739037:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.739045:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698773.739047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698773.739053:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146160512:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787698773.739061:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698773.739077:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146161280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698773.739087:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.739091:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698773.739093:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698773.739097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146160640:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698773.739101:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.739108:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.739112:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698773.739117:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698773.739131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698773.739133:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146161408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698773.739151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704f800 x1874531146160640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698773.739153:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698773.739161:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698773.739166:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698773.739176:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6673 00000100:00080000:2.0:1787698773.739183:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698773.739190:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698773.739196:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698773.739197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146160640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698773.739214:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146161536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698773.739219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146160640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698773.739242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146160768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698773.739263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704ce00 x1874531146160768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146160768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146160768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698773.739297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146160896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698773.739310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704c380 x1874531146160896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146160896/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146160896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698773.739341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146161024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698773.739394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704c000 x1874531146161024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146161024/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146161024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698773.739439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161152:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698773.739456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704d180 x1874531146161152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146161152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698773.739487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698773.739517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704fb80 x1874531146161280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146161280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698773.739564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698773.739577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704c700 x1874531146161408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146161408/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698773.739605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698773.739626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698773.739637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698773/real 1787698773] req@ffff8f518704df80 x1874531146161536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698773.739650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146161536/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698773.739659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698773.739663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698773.739666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146161536:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698774.634631:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698775.658507:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698776.682617:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698777.706600:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698778.730642:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698778.858477:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858490:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698778.858495:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858507:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858521:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858530:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698778.858573:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146161664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858602:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858605:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698778.858606:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858610:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858612:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858615:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858624:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698778.858635:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146161792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858642:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858645:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698778.858647:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858650:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858652:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858654:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698778.858666:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146161920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698778.858678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146161664:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698778.858685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698778.858690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858693:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858694:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858697:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698778.858711:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146162048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858718:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858721:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698778.858723:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858725:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858730:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858733:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858736:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698778.858741:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146162176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858749:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858751:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698778.858752:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858755:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858758:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858764:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858771:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698778.858772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698778.858786:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146162304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858793:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858796:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698778.858797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858802:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858804:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858806:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858809:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698778.858816:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146162432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858822:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858823:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698778.858825:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858829:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858830:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858833:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858835:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698778.858843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f518738bb80 x1874531146161664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698778.858844:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146162560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698778.858882:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698778.858886:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698778.858892:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698778.858900:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6678 00000100:00080000:3.0:1787698778.858905:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698778.858910:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698778.858914:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698778.858923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146161664/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698778.858930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146162688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698778.858957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.858964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.858969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146161664:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698778.858978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146161792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698778.859002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f518738b800 x1874531146161792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146161792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146161792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698778.859067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146161920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698778.859092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f5187388a80 x1874531146161920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146161920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146161920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698778.859138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146162048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698778.859155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f5187388380 x1874531146162048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146162048/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146162048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698778.859199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146162176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698778.859213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f518738aa00 x1874531146162176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146162176/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146162176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698778.859264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698778.859281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f5187389c00 x1874531146162304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146162304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698778.859323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698778.859340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f5187388e00 x1874531146162432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146162432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698778.859440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698778.859458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f518738ad80 x1874531146162560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146162560/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698778.859500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698778.859517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698778.859527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff8f5187389500 x1874531146162688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698778.859538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146162688/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.859546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698778.859549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698778.859552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146162688:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698779.754562:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698780.778514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698781.802536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698782.826531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698783.850518:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698783.978461:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978469:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698783.978473:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978480:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978489:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978494:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698783.978523:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146162816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698783.978543:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978545:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698783.978547:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978549:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978552:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978554:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978560:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698783.978568:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146162944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698783.978575:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978577:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698783.978578:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978582:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978585:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698783.978585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146162816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698783.978590:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978595:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698783.978601:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146163072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698783.978606:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978608:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698783.978610:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978612:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978614:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978616:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978618:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698783.978624:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146163200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698783.978629:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978631:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698783.978632:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978635:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978637:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978639:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978642:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698783.978646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698783.978652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146163328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698783.978659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698783.978662:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978665:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978667:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978669:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978671:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698783.978679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146163456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698783.978686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace91500 x1874531146162816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698783.978687:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698783.978696:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978702:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978705:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978710:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698783.978710:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146162816/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698783.978713:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698783.978723:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146163584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698783.978724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.978732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698783.978738:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698783.978739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146162816:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698783.978742:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698783.978743:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978746:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978751:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978754:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978757:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698783.978764:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146163712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698783.978770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146162944:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698783.978771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698783.978777:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698783.978780:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698783.978783:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6683 00000100:00080000:0.0:1787698783.978784:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698783.978786:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698783.978790:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698783.978795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698783.978799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146163840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698783.978807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace90380 x1874531146162944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.978817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146162944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.978823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.978826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.978828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146162944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698783.978832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146163072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698783.978848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.978855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace91880 x1874531146163072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.978861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146163072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.978866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.978868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.978870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146163072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698783.978873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146163200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698783.978883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.978888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace92d80 x1874531146163200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.978894:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146163200/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.978898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.978899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.978901:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146163200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698783.978904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146163328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698783.978912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.978917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace90e00 x1874531146163328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.978928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146163328/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.978932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.978933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.978935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146163328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698783.978938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698783.978953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.978959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace91180 x1874531146163456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.978965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146163456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.978970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.978972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.978973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698783.978976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698783.978989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.978995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace93480 x1874531146163584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.979001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146163584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.979009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.979011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.979012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698783.979023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698783.979032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.979038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace91c00 x1874531146163712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.979044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146163712/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.979048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.979049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.979051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698783.979054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698783.979062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698783.979067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff8f51ace92680 x1874531146163840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.979072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146163840/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.979076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.979078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698783.979080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146163840:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698784.874517:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698785.898498:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698786.922537:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698787.946551:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698788.970521:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698789.098436:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098445:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698789.098450:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698789.098459:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098470:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098477:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698789.098513:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146163968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698789.098543:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098545:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698789.098547:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698789.098550:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098553:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098555:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698789.098562:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698789.098571:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146164096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698789.098581:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698789.098585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698789.098591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146163968:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698789.098591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098598:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098604:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698789.098609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698789.098628:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146164224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698789.098642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698789.098647:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698789.098651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698789.098655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098660:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098664:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698789.098670:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698789.098682:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146164352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698789.098691:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098694:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698789.098698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704e680 x1874531146163968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698789.098698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698789.098707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098715:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098719:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698789.098722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146163968/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698789.098724:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698789.098735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698789.098739:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146164480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698789.098741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.098748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146163968:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698789.098757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698789.098759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098763:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698789.098765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698789.098771:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098776:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698789.098778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698789.098783:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698789.098789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704f480 x1874531146164096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698789.098789:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698789.098800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146164096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698789.098803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146164608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698789.098817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698789.098821:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698789.098823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698789.098825:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698789.098828:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698789.098829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164096:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698789.098835:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098839:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098841:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698789.098844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698789.098850:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146164736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698789.098856:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698789.098858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164224:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698789.098859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698789.098862:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698789.098867:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098869:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698789.098871:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698789.098875:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698789.098883:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146164864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698789.098890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.098902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704ed80 x1874531146164224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698789.098902:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698789.098910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698789.098915:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698789.098923:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6688 00000100:00080000:2.0:1787698789.098930:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698789.098935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146164224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698789.098937:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698789.098943:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698789.098947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.098952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698789.098956:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146164992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698789.098959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698789.098973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698789.098991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.099001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704dc00 x1874531146164352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698789.099012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146164352/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698789.099019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.099021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.099023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698789.099026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698789.099038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.099044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704ea00 x1874531146164480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698789.099050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146164480/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698789.099058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.099060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.099062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146164480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698789.099065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164608:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698789.099073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.099078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704e300 x1874531146164608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698789.099084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146164608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698789.099089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.099090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.099092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698789.099101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698789.099109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.099114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704d880 x1874531146164736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698789.099120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146164736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698789.099124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.099126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.099128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698789.099131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698789.099139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.099144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704d500 x1874531146164864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698789.099150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146164864/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698789.099154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.099155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.099157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698789.099159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698789.099167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698789.099172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698789/real 1787698789] req@ffff8f518704f480 x1874531146164992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698789.099178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146164992/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698789.099182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698789.099184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698789.099185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146164992:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698789.994502:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698791.018540:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698792.042529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698793.066501:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698794.090590:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698794.218445:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.218459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698794.218464:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.218475:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00080000:2.0:1787698794.218495:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.218505:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698794.218556:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146165120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698794.218602:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.218606:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698794.218609:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.218614:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00080000:2.0:1787698794.218618:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.218623:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698794.218634:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698794.218653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146165120:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698794.218655:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146165248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698794.218670:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.218674:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698794.218676:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.218680:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00080000:2.0:1787698794.218684:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.218688:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698794.218693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698794.218706:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146165376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698794.218715:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.218718:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698794.218720:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698794.218724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698794.218728:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00080000:2.0:1787698794.218734:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.218738:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698794.218743:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698794.218756:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146165504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698794.218772:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698794.218773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704e680 x1874531146165120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698794.218780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698794.218783:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.218792:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00080000:2.0:1787698794.218797:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698794.218804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146165120/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698794.218804:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698794.218812:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698794.218824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698794.218829:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146165632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698794.218831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.218839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146165120:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698794.218859:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.218864:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698794.218865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.218871:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00100000:0.0:1787698794.218874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698794.218876:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.218885:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698794.218891:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698794.218905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698794.218908:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146165760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698794.218920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704df80 x1874531146165248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698794.218921:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.218928:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698794.218930:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.218939:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00080000:2.0:1787698794.218945:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698794.218947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146165248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698794.218952:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698794.218960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787698794.218961:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698794.218967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.218972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698794.218977:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146165888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698794.218983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165376:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698794.218996:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.219000:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698794.219002:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698794.219007:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6693 00000800:00000100:0.0:1787698794.219011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698794.219014:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.219020:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698794.219025:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698794.219026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704c700 x1874531146165376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698794.219040:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146166016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698794.219044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146165376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698794.219062:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698794.219068:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698794.219069:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698794.219074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698794.219077:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6693 00000100:00100000:0.0:1787698794.219082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165376:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698794.219084:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698794.219092:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165504:192.168.201.125@tcp:8:kworker.0 00000008:00100000:2.0:1787698794.219099:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698794.219114:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146166144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698794.219124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704fb80 x1874531146165504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698794.219160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146165504/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698794.219173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.219177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698794.219183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698794.219202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704d180 x1874531146165632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698794.219223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146165632/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698794.219240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.219243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146165632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698794.219259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146165760:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698794.219273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704c000 x1874531146165760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698794.219287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146165760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698794.219306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.219309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146165760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698794.219314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146165888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698794.219333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704c380 x1874531146165888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698794.219400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146165888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698794.219408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.219411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146165888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698794.219415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146166016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698794.219429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704ce00 x1874531146166016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698794.219443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146166016/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698794.219449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.219451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146166016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698794.219454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146166144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698794.219461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698794.219466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698794/real 1787698794] req@ffff8f518704f800 x1874531146166144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698794.219472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146166144/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698794.219477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698794.219478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698794.219480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146166144:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698795.114524:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698796.138577:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698797.162605:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698798.186577:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698799.210582:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698799.338530:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338540:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698799.338544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698799.338553:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338563:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338571:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698799.338602:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146166272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698799.338627:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338629:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698799.338631:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698799.338634:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338636:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338639:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338648:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698799.338659:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146166400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698799.338669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338673:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698799.338675:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698799.338681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146166272:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698799.338681:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338687:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338693:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338697:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698799.338710:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146166528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698799.338720:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338723:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698799.338725:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698799.338728:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338732:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338736:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698799.338742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338742:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698799.338757:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146166656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698799.338767:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338770:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698799.338773:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698799.338785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f518738a300 x1874531146166272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698799.338787:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338794:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338801:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698799.338814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146166272/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698799.338822:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146166784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698799.338829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698799.338830:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338834:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698799.338837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698799.338840:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698799.338845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146166272:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698799.338848:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338853:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338856:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338859:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698799.338860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698799.338867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146166912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698799.338874:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338876:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698799.338878:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698799.338880:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338883:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338884:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338887:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698799.338889:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698799.338897:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146167040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698799.338904:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338907:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698799.338908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f5187388000 x1874531146166400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698799.338909:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698799.338915:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338919:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338924:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338931:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698799.338937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146166400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698799.338945:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146167168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698799.338949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.338956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.338963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166400:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698799.338964:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698799.338971:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698799.338973:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698799.338977:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6698 00000100:00080000:3.0:1787698799.338980:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698799.338983:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698799.338986:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698799.338994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698799.338997:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146167296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698799.339022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f518738b480 x1874531146166528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146166528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698799.339095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698799.339114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f5187389f80 x1874531146166656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146166656/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698799.339155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698799.339170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f518738a680 x1874531146166784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146166784/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146166784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698799.339225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146166912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698799.339243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f5187389180 x1874531146166912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146166912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146166912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698799.339292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146167040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698799.339314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f518738b100 x1874531146167040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146167040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146167040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698799.339401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146167168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698799.339420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f5187389500 x1874531146167168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146167168/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146167168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698799.339474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146167296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698799.339490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698799.339508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698799/real 1787698799] req@ffff8f518738ad80 x1874531146167296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698799.339520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146167296/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698799.339528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698799.339532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698799.339535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146167296:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698800.234593:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698801.258576:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698802.282576:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698803.306519:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698804.330608:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698804.458475:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458488:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698804.458493:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458514:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458528:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458537:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698804.458583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146167424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698804.458615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458619:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698804.458621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458626:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458630:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458634:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698804.458661:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146167552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698804.458663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146167424:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698804.458678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698804.458683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458688:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458691:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458695:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458700:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698804.458712:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146167680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698804.458722:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458724:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698804.458725:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458728:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6704 00000800:00000100:0.0:1787698804.458732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698804.458732:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458738:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458742:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698804.458751:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146167808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698804.458756:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458758:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698804.458760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458762:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458764:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458766:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458769:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698804.458778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f5187388e00 x1874531146167424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698804.458784:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146167936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698804.458790:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458793:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698804.458794:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458798:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458801:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698804.458803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146167424/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698804.458807:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458811:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698804.458817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698804.458821:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146168064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698804.458824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.458832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146167424:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698804.458834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458838:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698804.458839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458844:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458846:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458849:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698804.458854:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146168192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698804.458856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698804.458868:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698804.458870:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698804.458871:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698804.458874:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00080000:3.0:1787698804.458877:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698804.458882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698804.458883:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698804.458889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698804.458897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f5187389c00 x1874531146167552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698804.458898:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146168320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698804.458920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146167552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698804.458928:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698804.458932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698804.458933:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698804.458937:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698804.458940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698804.458945:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6704 00000100:00100000:0.0:1787698804.458948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167552:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698804.458951:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698804.458959:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698804.458961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167680:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698804.458965:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698804.458992:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146168448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698804.459001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f518738aa00 x1874531146167680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146167680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698804.459058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698804.459084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f5187388380 x1874531146167808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146167808/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698804.459122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698804.459135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f5187388a80 x1874531146167936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146167936/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146167936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698804.459172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698804.459190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f518738b800 x1874531146168064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146168064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698804.459232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698804.459244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f518738bb80 x1874531146168192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146168192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698804.459268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698804.459276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f5187388e00 x1874531146168320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146168320/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698804.459297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698804.459304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698804.459309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698804/real 1787698804] req@ffff8f518738ad80 x1874531146168448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698804.459314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146168448/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698804.459318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698804.459320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698804.459322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146168448:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698805.354567:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698806.378536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698807.402526:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698808.426577:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698809.450538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698809.578494:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578505:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698809.578509:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578518:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578527:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578534:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698809.578569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146168576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578604:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578608:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698809.578610:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578614:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578617:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578621:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698809.578642:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146168704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698809.578656:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578659:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578662:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578666:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698809.578680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146168832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578692:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698809.578693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578697:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578700:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578703:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698809.578717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146168960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578728:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698809.578730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578733:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578736:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578739:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578743:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698809.578754:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146169088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578762:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578765:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698809.578767:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578771:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578774:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578777:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698809.578789:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146169216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578797:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698809.578802:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578804:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578808:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578811:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578816:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698809.578826:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146169344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578834:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698809.578838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578841:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578844:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698809.578861:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146169472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698809.578869:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698809.578872:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698809.578874:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698809.578877:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6709 00000100:00080000:0.0:1787698809.578880:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698809.578883:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698809.578899:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698809.578909:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146169600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698809.578995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146168576:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787698809.579051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace93b80 x1874531146168576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146168576/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146168576:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698809.579126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146168704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698809.579147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace92300 x1874531146168704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146168704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146168704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698809.579187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146168832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698809.579204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace93100 x1874531146168832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146168832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146168832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698809.579243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146168960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698809.579261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace90700 x1874531146168960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146168960/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146168960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698809.579301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146169088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698809.579317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace90000 x1874531146169088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146169088/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146169088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698809.579378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698809.579393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace93800 x1874531146169216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146169216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698809.579419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169344:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698809.579428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace92680 x1874531146169344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146169344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698809.579450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698809.579458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace91c00 x1874531146169472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146169472/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698809.579479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698809.579486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698809.579491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698809/real 1787698809] req@ffff8f51ace93480 x1874531146169600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698809.579497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146169600/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698809.579501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698809.579502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698809.579504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146169600:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698810.474511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698811.498552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698812.522532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698813.546547:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698814.570483:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698814.698459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698814.698471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698814.698481:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698491:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698498:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698814.698529:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146169728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698814.698551:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698553:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698814.698555:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698814.698558:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698561:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698563:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698814.698571:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698814.698579:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146169856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698814.698586:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698588:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698814.698591:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698814.698592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146169728:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698814.698595:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698599:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698814.698605:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698814.698613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146169984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698814.698618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698814.698621:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698814.698624:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698625:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698628:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698814.698630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698814.698637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146170112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698814.698642:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698644:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698814.698645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698814.698648:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698650:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698651:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698814.698654:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698814.698659:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146170240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698814.698665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698667:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698814.698669:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698814.698674:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698677:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698681:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698814.698685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146169856:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698814.698687:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698814.698703:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146170368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698814.698710:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146169984:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698814.698716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698721:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698814.698723:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698814.698727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146170112:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698814.698729:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698736:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698741:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698814.698746:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698814.698746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146170240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698814.698761:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146170496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698814.698767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170368:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698814.698770:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698814.698775:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698814.698779:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698782:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698785:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698814.698788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170496:192.168.201.125@tcp:38:kworker.0 00000008:00100000:0.0:1787698814.698789:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698814.698818:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146170624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698814.698831:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698814.698835:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698814.698837:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698814.698841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170624:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698814.698842:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6714 00000100:00080000:0.0:1787698814.698850:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698814.698855:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698814.698861:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698814.698874:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146170752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698814.698913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170752:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698815.594496:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698816.618546:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698817.642509:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698817.770536:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787698817.770570:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698817.770582:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698817.770632:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787698817.770645:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698817.770654:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787698817.770665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace92680 x1874531146170752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698817.770666:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698817.770677:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698817.770688:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787698817.770701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146170752/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698817.770707:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787698817.770719:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:2.0:1787698817.770725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698817.770745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace91c00 x1874531146170624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146170624/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698817.770796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace93480 x1874531146170496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146170496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698817.770831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace91500 x1874531146170368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146170368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146170368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698817.770866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace90e00 x1874531146169856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146169856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146169856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698817.770903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace92d80 x1874531146169984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146169984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146169984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698817.770930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace91880 x1874531146170112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146170112/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770946:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146170112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698817.770956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace90380 x1874531146170240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146170240/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.770974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.770976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146170240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698817.770982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698814/real 1787698817] req@ffff8f51ace91180 x1874531146169728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698817.770992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146169728/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698817.770998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698817.771001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698817.771004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146169728:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787698818.666543:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698819.690502:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698819.818468:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818475:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698819.818478:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818485:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818493:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698819.818497:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698819.818525:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146170880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698819.818554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698819.818560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818564:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818567:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698819.818570:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698819.818579:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698819.818592:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146171008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698819.818602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146170880:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698819.818605:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698819.818620:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818624:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818628:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698819.818632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698819.818637:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698819.818649:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146171136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698819.818658:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698819.818662:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818665:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818667:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698819.818671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698819.818672:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698819.818679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698819.818691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146171264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698819.818700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818703:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698819.818705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818710:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818724:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698819.818731:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698819.818732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704f800 x1874531146170880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698819.818737:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698819.818752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146171392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698819.818760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146170880/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698819.818764:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818771:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698819.818775:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698819.818780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698819.818781:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818788:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698819.818791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698819.818796:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698819.818800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146170880:192.168.201.125@tcp:250:kworker.0 00000008:00100000:2.0:1787698819.818803:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698819.818820:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146171520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698819.818831:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698819.818834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171008:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698819.818836:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698819.818839:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818844:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818847:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698819.818851:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698819.818855:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698819.818867:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146171648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698819.818876:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698819.818880:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698819.818882:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818886:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:2.0:1787698819.818890:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698819.818894:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698819.818900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:2.0:1787698819.818900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698819.818917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704ce00 x1874531146171008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698819.818921:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146171776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698819.818943:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698819.818946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146171008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698819.818947:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698819.818952:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698819.818960:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6719 00000100:00080000:0.0:1787698819.818964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698819.818966:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698819.818972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698819.818973:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698819.818981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171008:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787698819.818981:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698819.818992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698819.818997:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146171904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698819.819019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704c380 x1874531146171136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146171136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698819.819064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698819.819083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704c000 x1874531146171264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146171264/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698819.819126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698819.819144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704d180 x1874531146171392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146171392/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146171392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698819.819212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698819.819231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704fb80 x1874531146171520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146171520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698819.819273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698819.819290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704c700 x1874531146171648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146171648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698819.819333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698819.819380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704df80 x1874531146171776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146171776/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698819.819431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698819.819448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698819.819459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698819/real 1787698819] req@ffff8f518704e680 x1874531146171904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698819.819470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146171904/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698819.819477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698819.819480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698819.819483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146171904:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698820.714487:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698821.738511:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698822.762513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698823.786567:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698824.810573:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698824.938511:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938518:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698824.938520:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938527:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938536:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698824.938541:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698824.938570:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146172032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698824.938592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938595:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698824.938596:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938600:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938603:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698824.938606:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938615:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698824.938639:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146172160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698824.938650:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698824.938652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146172032:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698824.938654:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698824.938659:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938665:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938669:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698824.938672:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938676:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698824.938693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146172288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698824.938699:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698824.938705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698824.938705:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938711:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938715:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698824.938718:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938722:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698824.938729:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146172416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698824.938735:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938737:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698824.938738:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698824.938742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f5187389500 x1874531146172032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698824.938742:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938744:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698824.938749:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938754:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698824.938765:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146172544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698824.938771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146172032/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698824.938772:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938779:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698824.938781:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938788:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938792:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698824.938795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698824.938795:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938801:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698824.938805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698824.938810:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146172672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698824.938811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146172032:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698824.938823:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938826:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698824.938827:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938831:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938835:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698824.938836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172160:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698824.938842:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938848:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698824.938858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698824.938862:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146172800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698824.938871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f518738b100 x1874531146172160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698824.938877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938881:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698824.938882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146172160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698824.938883:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698824.938891:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:0.0:1787698824.938894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698824.938895:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698824.938900:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698824.938902:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698824.938908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172160:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698824.938908:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698824.938919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698824.938922:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146172928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698824.938939:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698824.938941:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698824.938943:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698824.938947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698824.938950:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6724 00000100:00080000:3.0:1787698824.938954:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698824.938957:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698824.938960:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698824.938961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f5187389180 x1874531146172288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698824.938969:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146173056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698824.938973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146172288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.938988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.938991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.938994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698824.938998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698824.939011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698824.939018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f518738a680 x1874531146172416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698824.939025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146172416/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.939029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.939031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.939033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698824.939036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698824.939043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698824.939048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f5187389f80 x1874531146172544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698824.939058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146172544/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.939063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.939064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.939066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146172544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698824.939068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146172672:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698824.939076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698824.939081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f518738b480 x1874531146172672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698824.939088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146172672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.939092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.939094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.939095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146172672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698824.939106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146172800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698824.939114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698824.939119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f5187388000 x1874531146172800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698824.939126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146172800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.939142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.939144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.939145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146172800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698824.939149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146172928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698824.939158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698824.939164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f518738a300 x1874531146172928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698824.939170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146172928/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.939174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.939176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.939177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146172928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698824.939180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146173056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698824.939188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698824.939193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff8f5187389500 x1874531146173056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698824.939199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146173056/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698824.939203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698824.939204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698824.939206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146173056:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698825.834554:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698826.858552:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698827.882552:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698828.906570:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698829.930591:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698830.058452:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698830.058462:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058469:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058477:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058482:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698830.058511:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146173184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058530:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058533:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698830.058534:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058537:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058551:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058554:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058561:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698830.058570:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146173312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058578:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698830.058582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146173184:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698830.058582:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698830.058585:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058592:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058595:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058598:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698830.058606:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146173440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058611:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058614:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698830.058615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058617:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058639:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058642:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058646:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698830.058650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698830.058660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146173568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058680:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698830.058682:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058685:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058688:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058691:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058695:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698830.058698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183647800 x1874531146173184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698830.058708:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146173696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058717:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058720:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698830.058721:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698830.058723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146173184/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698830.058726:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058731:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058737:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698830.058739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698830.058740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698830.058745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.058749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146173184:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698830.058752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146173824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058758:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058760:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698830.058761:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058763:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058765:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058768:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058770:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698830.058771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698830.058780:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146173952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698830.058788:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698830.058790:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698830.058791:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698830.058794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787698830.058796:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058801:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058803:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058806:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698830.058809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183646680 x1874531146173312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698830.058816:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146174080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698830.058822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146173312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.058831:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.058833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698830.058833:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698830.058837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173312:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698830.058845:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698830.058847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173440:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698830.058852:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698830.058857:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6729 00000100:00080000:1.0:1787698830.058860:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698830.058864:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698830.058866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787698830.058872:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698830.058877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183644000 x1874531146173440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.058885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146173440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698830.058890:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146174208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698830.058892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.058898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.058904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698830.058914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698830.058925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698830.058932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183644700 x1874531146173568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.058939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146173568/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.058943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.058944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.058946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698830.058949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698830.058957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698830.058962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183647480 x1874531146173696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.058969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146173696/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.058973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.058974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.058976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146173696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698830.058978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146173824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698830.058986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698830.058991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183645c00 x1874531146173824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.058997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146173824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.059001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.059003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.059005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146173824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698830.059015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146173952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698830.059024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698830.059030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183645180 x1874531146173952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.059036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146173952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.059044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.059045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.059047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146173952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698830.059050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146174080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698830.059058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698830.059062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183647100 x1874531146174080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.059068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146174080/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.059072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.059074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.059075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146174080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698830.059078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146174208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698830.059086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698830.059090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698830/real 1787698830] req@ffff8f5183644380 x1874531146174208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698830.059096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146174208/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698830.059101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698830.059102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698830.059105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146174208:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698830.954496:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698831.978521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698833.002528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698834.026466:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698835.050511:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698835.178465:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178474:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698835.178477:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178487:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178497:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698835.178504:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698835.178537:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146174336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698835.178564:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698835.178569:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178574:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178578:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698835.178582:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698835.178591:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698835.178598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146174336:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698835.178606:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146174464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698835.178618:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698835.178623:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178627:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178631:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698835.178634:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698835.178639:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698835.178651:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146174592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698835.178660:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178664:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698835.178665:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178669:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178672:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698835.178675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698835.178678:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698835.178685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698835.178695:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146174720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698835.178704:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178707:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698835.178709:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178713:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178717:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698835.178721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace93800 x1874531146174336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698835.178723:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698835.178731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698835.178746:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146174848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698835.178748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146174336/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.178763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.178771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698835.178777:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178780:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698835.178783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698835.178788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146174336:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698835.178791:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178798:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698835.178802:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698835.178807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698835.178817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698835.178819:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146174976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698835.178832:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178835:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698835.178838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178845:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698835.178846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698835.178851:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698835.178857:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698835.178864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace90000 x1874531146174464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698835.178869:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146175104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698835.178890:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146174464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698835.178892:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178902:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698835.178905:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178910:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6734 00000100:00080000:0.0:1787698835.178915:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698835.178917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698835.178921:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698835.178926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:0.0:1787698835.178930:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698835.178932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698835.178941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698835.178944:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146175232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698835.178955:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698835.178959:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698835.178961:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698835.178966:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6734 00000800:00000100:2.0:1787698835.178966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698835.178973:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698835.178978:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698835.178985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace90700 x1874531146174592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698835.178996:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698835.178999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146174592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698835.179011:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146175360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698835.179021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698835.179040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698835.179061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698835.179072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace93100 x1874531146174720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698835.179083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146174720/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.179091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698835.179102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698835.179117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698835.179124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace92300 x1874531146174848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698835.179131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146174848/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.179136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146174848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698835.179152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146174976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698835.179167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698835.179178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace93b80 x1874531146174976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698835.179188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146174976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.179196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146174976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698835.179206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146175104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698835.179218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698835.179225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace93800 x1874531146175104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698835.179232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146175104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.179236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146175104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698835.179242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146175232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698835.179250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698835.179255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace91180 x1874531146175232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698835.179261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146175232/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.179266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146175232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698835.179271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146175360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698835.179279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698835.179284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698835/real 1787698835] req@ffff8f51ace90380 x1874531146175360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698835.179290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146175360/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698835.179295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698835.179296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698835.179298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146175360:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698836.074535:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698837.098554:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698838.122602:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698839.146549:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698840.170553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698840.298494:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298501:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698840.298503:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298510:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298519:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298525:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698840.298553:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146175488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298571:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298573:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698840.298574:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298577:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298579:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298580:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698840.298586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698840.298593:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146175616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298600:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298603:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698840.298605:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298609:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298613:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298617:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698840.298623:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698840.298623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146175488:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787698840.298638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146175744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298647:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298649:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698840.298650:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298653:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298656:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298658:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698840.298661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698840.298669:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146175872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298675:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298677:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698840.298678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298680:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298682:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298684:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698840.298686:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698840.298691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787698840.298697:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146176000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298703:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698840.298707:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298710:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298713:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298715:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698840.298718:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698840.298724:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146176128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298729:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698840.298732:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298734:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298737:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298739:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698840.298742:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698840.298746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183644e00 x1874531146175488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698840.298752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146176256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698840.298768:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298770:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698840.298771:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298775:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00080000:1.0:1787698840.298776:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298780:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698840.298780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146175488/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698840.298786:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698840.298795:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146176384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698840.298798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698840.298803:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698840.298807:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698840.298807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787698840.298811:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698840.298815:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6739 00000100:00100000:2.0:1787698840.298816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146175488:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787698840.298818:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698840.298825:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698840.298829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146175616:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787698840.298831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698840.298848:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146176512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698840.298859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.298875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183646d80 x1874531146175616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.298884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146175616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.298889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.298891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.298893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146175616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698840.298911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146175744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698840.298923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.298929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183646a00 x1874531146175744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.298935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146175744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.298940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.298941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.298943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146175744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698840.298946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146175872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698840.298955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.298960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183647b80 x1874531146175872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.298966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146175872/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.298971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.298972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.298974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146175872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698840.298976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146176000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698840.298985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.298990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183644380 x1874531146176000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.298996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146176000/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.299000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.299001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.299003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146176000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698840.299006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176128:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698840.299014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.299019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183647100 x1874531146176128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.299025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146176128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.299031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.299033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.299036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698840.299041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698840.299056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.299067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183645180 x1874531146176256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.299078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146176256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.299085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.299088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.299090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698840.299095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698840.299107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.299114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183645c00 x1874531146176384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.299120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146176384/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.299140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.299143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.299144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698840.299148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698840.299158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698840.299164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698840/real 1787698840] req@ffff8f5183647480 x1874531146176512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698840.299171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146176512/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698840.299175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698840.299181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698840.299183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146176512:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698841.194513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698842.218512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698843.242496:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698844.266510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698845.290498:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698845.418470:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418477:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698845.418480:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418489:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418497:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698845.418503:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698845.418533:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146176640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418550:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418552:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698845.418554:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418556:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418558:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698845.418560:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418567:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698845.418574:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146176768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418585:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418587:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698845.418588:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418591:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418593:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698845.418594:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418597:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698845.418604:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146176896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418610:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418612:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698845.418613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418615:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418617:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698845.418619:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418629:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698845.418644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146176640:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698845.418644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146177024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418669:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698845.418671:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418677:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418680:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698845.418683:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418688:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698845.418698:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146177152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698845.418716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418719:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418722:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698845.418728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698845.418730:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698845.418753:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146177280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418763:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418767:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698845.418768:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418775:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418782:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698845.418788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f518738ad80 x1874531146176640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698845.418789:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418797:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698845.418815:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146177408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698845.418827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698845.418831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146176640/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698845.418832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698845.418838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418854:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418862:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698845.418865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698845.418866:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698845.418879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698845.418879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698845.418891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146176640:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698845.418896:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146177536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698845.418905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146176768:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698845.418906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698845.418922:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698845.418926:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698845.418932:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6745 00000100:00080000:3.0:1787698845.418938:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698845.418943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698845.418947:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698845.418955:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698845.418959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f5187388e00 x1874531146176768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.418983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146176768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698845.418983:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146177664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698845.419044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146176768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698845.419059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146176896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698845.419074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f518738bb80 x1874531146176896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146176896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146176896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698845.419121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146177024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698845.419133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f518738b800 x1874531146177024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146177024/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146177024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698845.419159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146177152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698845.419168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f5187388a80 x1874531146177152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146177152/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146177152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698845.419192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177280:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698845.419201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f5187388380 x1874531146177280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146177280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698845.419229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698845.419239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f518738aa00 x1874531146177408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146177408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698845.419261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698845.419270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f5187389c00 x1874531146177536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146177536/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698845.419320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698845.419333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698845.419340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698845/real 1787698845] req@ffff8f5187389500 x1874531146177664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698845.419351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146177664/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698845.419392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698845.419394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698845.419397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146177664:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698846.314538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698847.338579:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698848.362543:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698849.386525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698850.410565:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698850.538473:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538487:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698850.538492:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538505:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538520:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538529:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698850.538572:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146177792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698850.538602:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538606:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698850.538608:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538613:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538616:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538621:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698850.538643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146177920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698850.538651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538653:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698850.538654:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538658:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538662:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698850.538663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146177792:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698850.538665:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538674:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698850.538682:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146178048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698850.538688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538691:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698850.538692:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538695:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538697:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538699:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538702:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698850.538709:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146178176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698850.538714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698850.538718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538720:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538737:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538739:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538743:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698850.538749:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146178304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698850.538757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698850.538758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538762:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698850.538763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538766:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538768:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538771:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538775:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698850.538782:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146178432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698850.538788:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538791:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698850.538791:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538794:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538796:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538798:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698850.538808:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146178560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698850.538814:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698850.538815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f518738a300 x1874531146177792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698850.538817:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698850.538820:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538827:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538830:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538832:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538836:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698850.538839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146177792/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698850.538846:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146178688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698850.538851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.538856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.538860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146177792:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698850.538860:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698850.538868:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698850.538870:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698850.538873:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6750 00000100:00080000:3.0:1787698850.538876:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698850.538879:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698850.538881:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698850.538887:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146178816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698850.538892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146177920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698850.538915:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.538926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f5187388000 x1874531146177920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.538934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146177920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.538939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.538941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.538943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146177920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698850.538947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146178048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698850.538964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.538973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f518738b480 x1874531146178048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.538980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146178048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.538985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.538987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.538989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146178048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698850.538992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146178176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698850.539003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.539008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f5187389f80 x1874531146178176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.539014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146178176/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.539019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.539021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.539023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146178176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698850.539026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146178304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698850.539034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.539040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f518738a680 x1874531146178304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.539046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146178304/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.539052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.539056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.539059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146178304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698850.539064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178432:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698850.539082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.539093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f5187389180 x1874531146178432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.539104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146178432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.539113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.539116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.539120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698850.539126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698850.539145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.539153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f518738b100 x1874531146178560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.539160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146178560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.539165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.539167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.539170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698850.539176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698850.539205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.539217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f5187389500 x1874531146178688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.539229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146178688/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.539234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.539236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.539238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698850.539253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698850.539269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698850.539280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698850/real 1787698850] req@ffff8f5187389c00 x1874531146178816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698850.539291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146178816/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698850.539311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698850.539313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698850.539317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146178816:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698851.434552:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698852.458594:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698853.482487:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698854.506568:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698855.530551:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698855.658519:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658537:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698855.658543:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698855.658555:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658570:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658577:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698855.658624:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146178944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698855.658656:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658660:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698855.658662:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698855.658667:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658671:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658675:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.658686:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698855.658703:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146179072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698855.658715:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658719:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698855.658722:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698855.658730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146178944:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698855.658732:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658739:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658752:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.658757:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698855.658768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146179200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698855.658780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658782:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698855.658784:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698855.658788:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658791:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658793:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.658796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698855.658803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146179328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698855.658809:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658812:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698855.658815:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698855.658820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698855.658824:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658829:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658832:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.658835:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698855.658846:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146179456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698855.658853:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658856:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698855.658857:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698855.658860:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658863:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658866:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.658870:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698855.658881:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146179584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698855.658887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace91880 x1874531146178944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698855.658891:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658901:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698855.658907:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698855.658922:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658929:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658935:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.658937:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698855.658945:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146179712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698855.658954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146178944/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698855.658955:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.658963:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698855.658967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698855.658978:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.658987:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.658996:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698855.658996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698855.659006:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698855.659007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146178944:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787698855.659024:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146179840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698855.659034:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698855.659036:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698855.659039:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698855.659043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179072:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698855.659044:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6755 00000100:00080000:0.0:1787698855.659054:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698855.659059:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698855.659063:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698855.659075:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146179968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698855.659080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace92d80 x1874531146179072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146179072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698855.659149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698855.659163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace90e00 x1874531146179200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146179200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698855.659189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698855.659198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace91500 x1874531146179328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146179328/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698855.659221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698855.659230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace93480 x1874531146179456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146179456/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146179456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698855.659252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698855.659260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace91c00 x1874531146179584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146179584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698855.659283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698855.659291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace92680 x1874531146179712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146179712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698855.659314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698855.659322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace90380 x1874531146179840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146179840/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698855.659385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698855.659396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698855.659403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698855/real 1787698855] req@ffff8f51ace91180 x1874531146179968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698855.659409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146179968/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698855.659414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698855.659415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698855.659417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146179968:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698856.554621:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698857.578586:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698858.602590:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698859.626557:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698860.650589:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698860.778452:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778466:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698860.778471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778484:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778497:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778504:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698860.778546:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146180096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698860.778580:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778584:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698860.778586:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778591:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778595:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.778612:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698860.778624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146180096:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698860.778629:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146180224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698860.778645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698860.778650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778654:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778657:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778661:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.778666:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698860.778679:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146180352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698860.778688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778691:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698860.778693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778696:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6760 00000800:00000100:0.0:1787698860.778701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698860.778702:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778709:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.778714:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698860.778728:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146180480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698860.778737:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698860.778741:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778746:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778750:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698860.778755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f518738aa00 x1874531146180096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698860.778757:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.778767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698860.778778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146180096/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698860.778783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146180608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698860.778791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698860.778796:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698860.778797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.778804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146180096:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698860.778804:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698860.778808:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778815:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778818:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778822:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.778827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698860.778832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698860.778840:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146180736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698860.778851:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778856:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698860.778857:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698860.778862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698860.778863:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778869:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778872:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.778875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698860.778884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146180864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698860.778894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f5187388380 x1874531146180224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698860.778901:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778908:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698860.778913:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698860.778920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146180224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698860.778920:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778925:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698860.778932:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698860.778937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698860.778939:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698860.778942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.778949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698860.778956:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146180992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698860.778958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180352:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698860.778973:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698860.778976:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698860.778979:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698860.778983:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6760 00000100:00080000:3.0:1787698860.778987:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698860.778991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698860.778994:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698860.779001:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698860.779013:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146181120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698860.779016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f5187388a80 x1874531146180352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146180352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698860.779068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698860.779087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698860.779108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f518738b800 x1874531146180480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146180480/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698860.779140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698860.779159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698860.779169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f518738bb80 x1874531146180608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146180608/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146180608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698860.779206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146180736:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698860.779221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698860.779231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f5187388e00 x1874531146180736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146180736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146180736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698860.779260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146180864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698860.779277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698860.779286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f518738ad80 x1874531146180864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146180864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146180864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698860.779313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146180992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698860.779330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698860.779337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f518738aa00 x1874531146180992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146180992/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146180992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698860.779379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146181120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698860.779392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698860.779401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698860/real 1787698860] req@ffff8f5187389c00 x1874531146181120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698860.779408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146181120/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698860.779413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698860.779414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698860.779416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146181120:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698861.674609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698862.698572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698863.722524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698864.746506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698865.770498:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698865.898441:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898449:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698865.898452:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.898459:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898468:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698865.898473:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698865.898505:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146181248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698865.898523:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898525:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698865.898527:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.898530:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898531:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698865.898533:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898540:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698865.898547:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146181376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698865.898555:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898556:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698865.898557:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.898559:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898561:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698865.898563:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898566:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698865.898569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146181248:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698865.898575:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146181504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698865.898583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698865.898588:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.898591:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898594:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698865.898598:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898603:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698865.898613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146181632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698865.898626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698865.898633:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898636:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698865.898638:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.898643:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898646:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698865.898649:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898654:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698865.898660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f5187389500 x1874531146181248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698865.898667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146181760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698865.898677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146181248/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.898677:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898686:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698865.898690:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698865.898693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698865.898696:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:0.0:1787698865.898699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698865.898703:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698865.898704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146181248:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698865.898711:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898716:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698865.898724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698865.898727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146181888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698865.898738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787698865.898744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698865.898745:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.898751:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898755:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698865.898757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f518738b100 x1874531146181376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.898760:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898768:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698865.898780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146181376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698865.898783:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146182016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698865.898793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.898799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698865.898799:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698865.898803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181376:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698865.898806:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698865.898809:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698865.898810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181504:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698865.898816:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898820:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698865.898824:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698865.898828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698865.898836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f5187389180 x1874531146181504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698865.898840:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698865.898843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146181504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698865.898851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.898853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.898856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698865.898859:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146182144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698865.898862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181632:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698865.898875:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.898878:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698865.898880:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698865.898883:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698865.898889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f518738a680 x1874531146181632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.898890:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6765 00000100:00080000:3.0:1787698865.898896:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698865.898901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146181632/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.898904:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698865.898913:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698865.898915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.898919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.898921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698865.898926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698865.898944:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146182272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698865.898961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698865.898969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f5187389f80 x1874531146181760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698865.898977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146181760/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698865.898982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.898984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.898986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146181760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698865.898996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146181888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698865.899013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698865.899019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f518738b480 x1874531146181888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698865.899025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146181888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698865.899030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.899032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.899034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146181888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698865.899037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146182016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698865.899046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698865.899051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f5187388000 x1874531146182016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698865.899058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146182016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698865.899062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.899063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.899065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146182016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698865.899067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146182144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698865.899075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698865.899080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f518738a300 x1874531146182144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698865.899086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146182144/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698865.899090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.899091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.899093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146182144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698865.899095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146182272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698865.899103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698865.899108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff8f5187389500 x1874531146182272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698865.899114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146182272/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698865.899118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698865.899120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698865.899121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146182272:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698866.794593:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698867.818574:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698868.842522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698869.866495:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698870.890575:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698871.018530:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018541:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698871.018544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018554:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018566:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018572:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698871.018615:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146182400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698871.018645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698871.018648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018652:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018654:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018656:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.018664:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698871.018672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146182528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698871.018684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018687:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698871.018688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018690:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018693:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018695:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.018697:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698871.018707:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146182656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698871.018712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018714:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698871.018716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018719:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018724:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.018727:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698871.018735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146182400:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698871.018736:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146182784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698871.018758:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698871.018760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018765:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018772:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018774:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.018778:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698871.018788:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146182912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698871.018795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698871.018798:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018801:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018804:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018806:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.018810:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698871.018816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698871.018820:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146183040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698871.018827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.018830:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698871.018832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018836:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018839:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698871.018842:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698871.018878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f5187389c00 x1874531146182400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698871.018879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698871.018895:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146183168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698871.018901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146182400/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.018945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.018951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.018955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146182400:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698871.018957:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698871.018963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182528:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698871.018963:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698871.018969:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698871.018978:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:3.0:1787698871.018983:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698871.018992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698871.018992:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.019003:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698871.019008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f518738aa00 x1874531146182528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698871.019017:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146183296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698871.019051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146182528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698871.019051:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698871.019070:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698871.019077:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698871.019079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698871.019082:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6770 00000100:00080000:0.0:1787698871.019088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698871.019090:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698871.019095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182528:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698871.019098:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698871.019109:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698871.019114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698871.019124:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146183424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698871.019141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f518738ad80 x1874531146182656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146182656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698871.019208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698871.019223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f5187388e00 x1874531146182784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146182784/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698871.019252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698871.019261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f518738bb80 x1874531146182912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146182912/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146182912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698871.019285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698871.019301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f518738b800 x1874531146183040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146183040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698871.019340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698871.019419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f5187388a80 x1874531146183168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146183168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698871.019448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698871.019461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f5187388380 x1874531146183296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146183296/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698871.019483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698871.019491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698871.019496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698871/real 1787698871] req@ffff8f5187389500 x1874531146183424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698871.019511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146183424/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698871.019518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698871.019521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698871.019524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146183424:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698871.914598:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698872.938459:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698873.962508:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698874.986545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698876.010524:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698876.138498:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138511:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698876.138517:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138528:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138541:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138549:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698876.138587:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146183552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698876.138614:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138618:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698876.138620:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138624:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138627:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138631:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698876.138642:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698876.138654:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146183680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698876.138663:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138666:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698876.138667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138670:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138673:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138676:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698876.138680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146183552:192.168.201.125@tcp:250:kworker.0 00000008:00100000:0.0:1787698876.138685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698876.138699:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146183808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698876.138707:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138710:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698876.138712:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138715:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138718:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138720:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698876.138726:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698876.138734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146183936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698876.138742:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138745:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698876.138747:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138750:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138753:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138772:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787698876.138774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787698876.138777:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698876.138787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146184064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698876.138794:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698876.138799:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138803:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138806:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138809:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698876.138813:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698876.138824:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146184192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698876.138830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace93800 x1874531146183552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698876.138832:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138836:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698876.138838:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138841:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138844:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138847:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698876.138849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146183552/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698876.138851:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698876.138859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698876.138860:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146184320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698876.138863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.138867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146183552:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698876.138868:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698876.138873:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698876.138874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146183680:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698876.138877:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00080000:0.0:1787698876.138880:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138884:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698876.138888:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698876.138894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698876.138897:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146184448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698876.138905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace93b80 x1874531146183680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698876.138906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698876.138909:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698876.138911:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698876.138914:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6775 00000100:00100000:2.0:1787698876.138916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146183680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698876.138917:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698876.138920:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698876.138923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698876.138924:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698876.138926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.138929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146183680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698876.138932:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146184576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698876.138955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146183808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698876.138973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.138983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace92300 x1874531146183808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.138990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146183808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.138995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.138997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.138999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146183808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698876.139002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146183936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698876.139013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.139019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace93100 x1874531146183936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.139025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146183936/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.139029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.139031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.139032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146183936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698876.139035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146184064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698876.139044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.139049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace90700 x1874531146184064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.139056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146184064/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.139060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.139062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.139063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146184064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698876.139066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698876.139075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.139080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace90000 x1874531146184192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.139087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146184192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.139092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.139093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.139096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698876.139101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698876.139116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.139125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace91180 x1874531146184320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.139135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146184320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.139148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.139151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.139155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698876.139158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698876.139171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.139178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace90380 x1874531146184448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.139186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146184448/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.139193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.139196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.139198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698876.139203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698876.139220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698876.139230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698876/real 1787698876] req@ffff8f51ace92680 x1874531146184576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698876.139240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146184576/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698876.139245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698876.139247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698876.139249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146184576:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698877.034541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698878.058550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698879.082583:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698880.106595:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698881.130587:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698881.258521:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258540:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698881.258544:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258558:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258572:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258580:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787698881.258629:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146184704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258661:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258664:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698881.258666:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258680:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258684:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258694:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698881.258707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146184704:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787698881.258709:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698881.258728:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146184832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258737:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258740:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698881.258741:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258744:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258748:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258751:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.258755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698881.258763:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146184960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258771:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258775:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698881.258777:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258781:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258785:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258789:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.258794:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698881.258808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146185088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698881.258822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258827:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00100000:2.0:1787698881.258828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146184832:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698881.258830:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258837:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.258842:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698881.258855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146185216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258867:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698881.258869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146184960:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698881.258870:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698881.258874:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258879:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258882:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258885:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.258888:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698881.258902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146185088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698881.258903:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146185344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258915:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258918:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698881.258919:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258922:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258925:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258928:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.258931:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698881.258940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146185216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787698881.258943:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146185472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258952:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258955:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698881.258956:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698881.258959:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.258963:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.258967:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.258971:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698881.258972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787698881.258985:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146185600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698881.258995:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698881.258999:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698881.259001:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698881.259004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185472:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787698881.259009:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6780 00000100:00080000:1.0:1787698881.259013:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698881.259016:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787698881.259021:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698881.259031:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146185728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698881.259036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698881.259068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185728:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698882.154573:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698883.178562:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698884.202601:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698884.330758:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:0.0:1787698884.330816:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698884.330837:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787698884.330902:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:0.0:1787698884.330920:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787698884.330932:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787698884.330944:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787698884.330952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183644380 x1874531146185728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698884.330956:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787698884.330969:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787698884.330980:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787698884.330989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146185728/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698884.330994:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00080000:2.0:1787698884.331011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698884.331033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183645180 x1874531146185472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146185472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698884.331066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183647100 x1874531146185600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146185600/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698884.331107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183647480 x1874531146185216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146185216/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146185216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698884.331140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183645c00 x1874531146185344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146185344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146185344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698884.331173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183644000 x1874531146184832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146184832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146184832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698884.331204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183646680 x1874531146184960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146184960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146184960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698884.331235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183647800 x1874531146185088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146185088/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146185088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698884.331263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698881/real 1787698884] req@ffff8f5183644700 x1874531146184704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698884.331274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146184704/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698884.331282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698884.331284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698884.331293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146184704:192.168.201.125@tcp:250:kworker.0 00010000:00010000:1.0:1787698885.226576:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698886.250507:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698886.378436:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.378446:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698886.378450:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378461:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378472:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378479:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698886.378511:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146185856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698886.378547:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.378555:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698886.378557:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378573:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378577:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378607:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698886.378611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146185856:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698886.378617:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698886.378633:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146185984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698886.378644:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.378648:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698886.378650:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378654:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378659:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378674:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787698886.378675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698886.378683:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698886.378702:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146186112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698886.378713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.378718:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787698886.378719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f518738a300 x1874531146185856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698886.378720:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378728:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378735:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378741:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698886.378747:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698886.378749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146185856/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698886.378761:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146186240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698886.378762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.378771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698886.378778:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698886.378780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146185856:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698886.378783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698886.378786:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378792:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378796:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378800:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698886.378805:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698886.378809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146185984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698886.378820:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146186368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698886.378832:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698886.378837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698886.378839:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698886.378842:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378848:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00100000:2.0:1787698886.378854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f5187388000 x1874531146185984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698886.378858:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698886.378870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146185984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698886.378870:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698886.378879:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698886.378882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.378888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787698886.378894:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146186496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698886.378895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146185984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698886.378908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146186112:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698886.378910:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.378921:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698886.378922:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.378926:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378930:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378934:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698886.378938:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698886.378943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698886.378953:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146186624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698886.378959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f518738b480 x1874531146186112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698886.378970:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.378977:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698886.378979:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698886.378982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146186112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698886.378986:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6786 00000100:00080000:3.0:1787698886.378991:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.378997:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698886.379000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698886.379004:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698886.379007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146186112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698886.379018:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146186752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698886.379021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146186240:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698886.379037:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698886.379040:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698886.379043:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698886.379047:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6786 00000800:00000100:2.0:1787698886.379050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698886.379054:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698886.379061:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698886.379066:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698886.379067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f5187389f80 x1874531146186240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698886.379079:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146186880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698886.379085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146186240/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698886.379098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.379101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146186240:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698886.379121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146186368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698886.379139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698886.379150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f518738a680 x1874531146186368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698886.379162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146186368/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698886.379170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.379173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146186368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698886.379182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186496:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698886.379197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698886.379207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f5187389180 x1874531146186496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698886.379218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146186496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698886.379233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.379236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698886.379245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698886.379260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698886.379270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f518738b100 x1874531146186624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698886.379281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146186624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698886.379295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.379298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698886.379307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698886.379323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698886.379332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f518738a300 x1874531146186752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698886.379343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146186752/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698886.379350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.379352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698886.379393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698886.379411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698886.379421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698886/real 1787698886] req@ffff8f5187389500 x1874531146186880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698886.379432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146186880/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698886.379439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698886.379442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698886.379445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146186880:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698887.274612:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698888.298523:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698889.322851:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698890.346486:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698891.370540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698891.498468:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498478:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698891.498482:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498510:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498520:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.498527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698891.498569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146187008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698891.498597:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498601:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698891.498603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498611:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.498615:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698891.498631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698891.498646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146187008:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698891.498650:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146187136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698891.498662:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498665:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698891.498667:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498671:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498675:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.498678:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698891.498683:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698891.498701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146187264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698891.498703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698891.498714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498717:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698891.498719:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498724:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498728:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.498732:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698891.498737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698891.498745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704f480 x1874531146187008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698891.498750:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146187392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698891.498768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498773:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698891.498774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146187008/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698891.498775:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498783:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498790:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698891.498791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698891.498795:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698891.498799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787698891.498801:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698891.498813:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146187520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698891.498817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146187008:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698891.498829:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498832:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698891.498835:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498839:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498843:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698891.498847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187136:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698891.498850:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698891.498857:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698891.498870:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146187648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698891.498879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698891.498884:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698891.498889:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698891.498891:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698891.498895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704d500 x1874531146187136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698891.498897:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498903:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.498908:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698891.498914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146187136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698891.498915:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698891.498928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698891.498930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146187776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698891.498934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.498941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187136:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698891.498947:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698891.498951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187264:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698891.498952:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698891.498956:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698891.498961:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00080000:2.0:1787698891.498965:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.498969:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698891.498974:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698891.498975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698891.498986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704d880 x1874531146187264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698891.498992:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146187904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698891.499003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146187264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698891.499004:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698891.499011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698891.499015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.499018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698891.499024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698891.499038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698891.499047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704e300 x1874531146187392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698891.499056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146187392/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698891.499063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698891.499069:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698891.499070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698891.499072:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698891.499075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187392:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698891.499079:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6791 00000100:00100000:0.0:1787698891.499085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187520:192.168.201.125@tcp:8:kworker.0 00000100:00080000:2.0:1787698891.499085:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698891.499091:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698891.499098:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698891.499106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698891.499111:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146188032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698891.499120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704ea00 x1874531146187520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698891.499131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146187520/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698891.499139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698891.499143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.499146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146187520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698891.499168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146187648:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698891.499185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698891.499195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704dc00 x1874531146187648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698891.499204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146187648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698891.499212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698891.499214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.499218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146187648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698891.499223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146187776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698891.499235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698891.499243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704ed80 x1874531146187776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698891.499253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146187776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698891.499259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698891.499262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.499265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146187776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698891.499270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146187904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698891.499283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698891.499291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704f480 x1874531146187904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698891.499301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146187904/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698891.499314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698891.499317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.499320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146187904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698891.499324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146188032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698891.499338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698891.499347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698891/real 1787698891] req@ffff8f518704e680 x1874531146188032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698891.499391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146188032/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698891.499399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698891.499402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698891.499405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146188032:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698892.394551:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698893.418547:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698894.442573:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698895.466571:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698896.490552:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698896.618490:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618499:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698896.618503:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618512:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618522:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618527:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698896.618554:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146188160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618579:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698896.618580:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618583:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618585:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618587:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618594:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698896.618601:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146188288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618612:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698896.618615:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698896.618620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146188160:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698896.618621:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618625:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618628:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698896.618639:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146188416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618644:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698896.618648:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618650:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618652:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618654:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698896.618662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146188544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618667:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618669:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698896.618670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618672:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618674:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698896.618676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698896.618680:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618684:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698896.618691:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146188672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618696:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618698:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698896.618699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618702:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618704:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618706:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698896.618711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f5187388380 x1874531146188160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698896.618718:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146188800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698896.618728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146188160/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698896.618729:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698896.618731:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618737:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618739:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618742:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698896.618752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698896.618754:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146188928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698896.618760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.618768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146188160:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698896.618770:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618776:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698896.618777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618780:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618782:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698896.618785:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618788:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698896.618796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698896.618799:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146189056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698896.618811:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698896.618814:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698896.618817:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698896.618820:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6796 00000100:00080000:3.0:1787698896.618824:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698896.618826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698896.618831:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698896.618837:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698896.618842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f5187388a80 x1874531146188288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698896.618850:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146189184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698896.618862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146188288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.618868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.618871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.618873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698896.618879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698896.618893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.618899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f518738b800 x1874531146188416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.618905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146188416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.618915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.618917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.618920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698896.618925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698896.618943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.618953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f518738bb80 x1874531146188544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.618964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146188544/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.618973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.618975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.618977:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698896.618981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698896.618999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.619009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f5187388e00 x1874531146188672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.619020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146188672/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.619028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.619030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.619032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146188672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698896.619036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146188800:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698896.619049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.619055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f518738ad80 x1874531146188800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.619062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146188800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.619067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.619068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.619071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146188800:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698896.619075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146188928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698896.619091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.619101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f518738aa00 x1874531146188928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.619112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146188928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.619119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.619121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.619123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146188928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698896.619137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146189056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698896.619150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.619156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f5187389c00 x1874531146189056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.619162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146189056/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.619166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.619168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.619170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146189056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698896.619173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146189184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698896.619180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698896.619185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698896/real 1787698896] req@ffff8f5187388380 x1874531146189184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698896.619192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146189184/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698896.619196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698896.619197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698896.619199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146189184:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698897.514553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698898.538624:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698899.562600:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698900.586528:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698901.610540:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698901.738437:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738448:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698901.738452:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738462:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00080000:3.0:1787698901.738474:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698901.738481:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698901.738526:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146189312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698901.738558:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738561:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698901.738563:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738566:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00080000:3.0:1787698901.738569:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698901.738573:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.738581:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698901.738592:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146189440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698901.738606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738613:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698901.738614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146189312:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698901.738616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738622:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00080000:3.0:1787698901.738631:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698901.738635:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.738640:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698901.738652:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146189568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698901.738659:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738662:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698901.738663:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738665:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00080000:3.0:1787698901.738680:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698901.738682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698901.738687:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.738693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698901.738704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146189696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698901.738714:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698901.738717:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738722:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00100000:0.0:1787698901.738724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f5187389500 x1874531146189312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698901.738725:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698901.738731:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.738737:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698901.738748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146189312/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698901.738752:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146189824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698901.738769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698901.738771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698901.738783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698901.738788:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787698901.738789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146189312:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698901.738791:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738801:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00080000:3.0:1787698901.738805:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698901.738809:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.738815:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698901.738819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698901.738829:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146189952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698901.738840:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738844:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698901.738846:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738850:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6801 00000800:00000100:0.0:1787698901.738854:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698901.738856:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698901.738870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f518738a300 x1874531146189440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698901.738872:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.738879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698901.738894:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146190080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698901.738903:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738906:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698901.738908:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738912:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6801 00000100:00100000:0.0:1787698901.738916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146189440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698901.738918:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698901.738925:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698901.738930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698901.738932:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698901.738937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.738941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698901.738946:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146190208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698901.738950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698901.738963:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698901.738966:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698901.738969:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698901.738973:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6801 00000800:00000100:0.0:1787698901.738978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698901.738987:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698901.738993:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f518738b100 x1874531146189568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698901.738997:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698901.739006:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698901.739013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146189568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698901.739020:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146190336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698901.739038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698901.739063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698901.739083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698901.739094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f5187389180 x1874531146189696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698901.739106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146189696/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698901.739130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698901.739143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698901.739160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698901.739169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f518738a680 x1874531146189824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698901.739180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146189824/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698901.739187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146189824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698901.739209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146189952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698901.739226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698901.739235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f5187389f80 x1874531146189952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698901.739246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146189952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698901.739254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146189952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698901.739266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146190080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698901.739281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698901.739291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f518738b480 x1874531146190080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698901.739301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146190080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698901.739309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146190080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698901.739341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146190208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698901.739412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698901.739446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f5187388000 x1874531146190208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698901.739458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146190208/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698901.739466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146190208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698901.739479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146190336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698901.739496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698901.739506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698901/real 1787698901] req@ffff8f5187389500 x1874531146190336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698901.739516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146190336/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698901.739524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698901.739527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698901.739530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146190336:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698902.634470:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698903.658549:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698904.682587:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698905.706570:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698906.730569:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698906.858509:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858526:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698906.858530:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698906.858543:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858556:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858563:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698906.858609:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146190464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858650:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698906.858652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698906.858662:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858665:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858669:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698906.858679:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698906.858691:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146190592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858709:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698906.858711:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698906.858716:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858718:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858723:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698906.858724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146190464:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698906.858728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698906.858743:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146190720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858757:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698906.858759:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698906.858763:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858767:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858773:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698906.858779:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698906.858788:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146190848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858797:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858801:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698906.858803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698906.858808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698906.858809:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858815:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858820:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698906.858825:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698906.858837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146190976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858844:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858846:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698906.858848:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698906.858851:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858854:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858856:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698906.858860:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698906.858863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f5187388380 x1874531146190464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698906.858870:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146191104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858881:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858883:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698906.858884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698906.858887:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858889:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698906.858891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146190464/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698906.858893:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698906.858896:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698906.858904:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146191232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858925:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858927:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698906.858928:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698906.858930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698906.858930:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858934:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858939:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698906.858940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698906.858941:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698906.858950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146190464:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698906.858951:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146191360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698906.858962:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698906.858964:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698906.858966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698906.858967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190592:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698906.858970:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6806 00000100:00080000:3.0:1787698906.858974:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698906.858977:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698906.858979:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698906.858986:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146191488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698906.858999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f5187389c00 x1874531146190592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146190592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698906.859079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698906.859098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f518738aa00 x1874531146190720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146190720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698906.859136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698906.859153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f518738ad80 x1874531146190848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146190848/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698906.859196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698906.859212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f5187388e00 x1874531146190976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146190976/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146190976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698906.859265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698906.859283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f518738bb80 x1874531146191104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146191104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698906.859322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698906.859339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f518738b800 x1874531146191232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146191232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698906.859413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698906.859431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f5187388a80 x1874531146191360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146191360/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698906.859470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698906.859486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698906.859497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff8f5187389500 x1874531146191488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698906.859507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146191488/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.859514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698906.859516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698906.859519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146191488:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698907.754495:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698908.778568:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698909.802515:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698910.826479:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698911.850461:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698911.978455:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978463:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698911.978466:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978473:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978483:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698911.978489:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698911.978529:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146191616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698911.978554:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978557:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698911.978558:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978561:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978564:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698911.978567:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698911.978574:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698911.978583:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146191744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698911.978593:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978596:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787698911.978602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146191616:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698911.978603:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978610:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978615:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698911.978618:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698911.978623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698911.978637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146191872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698911.978645:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978648:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698911.978649:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978652:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978655:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698911.978657:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698911.978660:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698911.978669:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146192000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698911.978674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698911.978679:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978681:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698911.978683:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978686:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978689:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698911.978691:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698911.978693:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698911.978703:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146192128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698911.978712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f5187388000 x1874531146191616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698911.978713:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978721:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698911.978724:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978731:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00100000:0.0:1787698911.978735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146191616/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698911.978736:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698911.978744:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787698911.978747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698911.978751:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698911.978752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.978758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146191616:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698911.978768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146192256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698911.978779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146191744:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698911.978781:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978786:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698911.978788:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978792:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978795:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698911.978798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698911.978802:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698911.978807:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698911.978812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f518738b480 x1874531146191744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698911.978817:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146192384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698911.978844:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698911.978848:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698911.978850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146191744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698911.978855:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978864:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:0.0:1787698911.978866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698911.978869:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698911.978871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698911.978876:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698911.978877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146191744:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698911.978884:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698911.978885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146191872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698911.978898:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146192512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698911.978905:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698911.978909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698911.978910:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698911.978913:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698911.978916:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6811 00000100:00080000:3.0:1787698911.978920:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698911.978921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f5187389f80 x1874531146191872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698911.978924:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698911.978929:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698911.978936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146191872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698911.978939:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146192640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698911.978954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.978960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.978964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146191872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698911.978971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146192000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698911.978991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698911.979000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f518738a680 x1874531146192000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698911.979007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146192000/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.979015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.979017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.979019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146192000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698911.979022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146192128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698911.979033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698911.979038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f5187389180 x1874531146192128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698911.979044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146192128/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.979048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.979050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.979051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146192128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698911.979064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698911.979079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698911.979089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f518738b100 x1874531146192256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698911.979101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146192256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.979108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.979110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.979112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698911.979116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698911.979132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698911.979142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f518738a300 x1874531146192384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698911.979183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146192384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.979191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.979193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.979196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698911.979201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698911.979218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698911.979228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f5187388000 x1874531146192512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698911.979236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146192512/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.979242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.979244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.979246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698911.979251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698911.979265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698911.979274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff8f5187389500 x1874531146192640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698911.979284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146192640/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.979290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.979292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698911.979295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146192640:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698912.874532:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698913.898544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698914.922586:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698915.946575:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698916.970580:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698917.098541:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098556:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698917.098561:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.098575:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:0.0:1787698917.098589:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.098597:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698917.098648:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146192768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698917.098684:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098688:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698917.098690:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.098695:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:0.0:1787698917.098700:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.098704:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.098717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698917.098734:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146192896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698917.098740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146192768:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698917.098755:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698917.098761:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.098767:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:0.0:1787698917.098771:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.098775:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.098781:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698917.098794:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146193024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698917.098804:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098808:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698917.098810:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.098814:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:0.0:1787698917.098818:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.098821:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.098825:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698917.098833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698917.098837:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146193152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698917.098844:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098847:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698917.098850:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.098853:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:0.0:1787698917.098856:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.098859:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.098863:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698917.098874:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146193280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698917.098883:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098887:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698917.098890:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.098895:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:0.0:1787698917.098899:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698917.098901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace91c00 x1874531146192768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698917.098906:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.098914:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698917.098926:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146193408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698917.098929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146192768/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698917.098939:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.098946:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698917.098948:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698917.098951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698917.098952:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00080000:2.0:1787698917.098959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.098964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146192768:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698917.098973:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.098980:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.098985:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698917.099000:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146193536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698917.099003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146192896:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698917.099019:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.099023:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698917.099025:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698917.099031:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6816 00000800:00000100:2.0:1787698917.099035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698917.099043:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698917.099050:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace93480 x1874531146192896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698917.099057:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698917.099083:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146193664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698917.099091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146192896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698917.099097:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698917.099107:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698917.099108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698917.099109:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698917.099115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698917.099120:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6816 00000100:00100000:2.0:1787698917.099123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146192896:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698917.099126:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698917.099132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146193024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698917.099134:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698917.099142:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698917.099157:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146193792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698917.099161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace91500 x1874531146193024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146193024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146193024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698917.099216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146193152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698917.099232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace90e00 x1874531146193152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146193152/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146193152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698917.099270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146193280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698917.099284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace92d80 x1874531146193280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146193280/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146193280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698917.099322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698917.099338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace91880 x1874531146193408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146193408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698917.099419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698917.099437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace91c00 x1874531146193536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146193536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698917.099476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698917.099492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace92680 x1874531146193664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146193664/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698917.099529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698917.099544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698917.099552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698917/real 1787698917] req@ffff8f51ace90380 x1874531146193792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698917.099560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146193792/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698917.099567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698917.099569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698917.099572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146193792:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698917.994631:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698919.018525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698920.042531:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698921.066550:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698922.090515:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698922.218467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218474:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698922.218477:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218485:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218494:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218500:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698922.218528:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146193920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698922.218554:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698922.218560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218564:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218568:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218572:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218580:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698922.218594:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146194048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698922.218600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146193920:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698922.218617:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698922.218623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218627:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218631:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218634:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218639:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698922.218652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146194176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698922.218656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698922.218664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698922.218669:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218672:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218675:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218677:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698922.218689:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146194304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698922.218694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f5187388a80 x1874531146193920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698922.218700:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218706:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698922.218709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218713:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218716:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698922.218718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146193920/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698922.218721:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218726:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698922.218735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698922.218737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146194432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698922.218741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.218748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146193920:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698922.218751:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218755:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698922.218756:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218759:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218762:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218763:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218766:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698922.218771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698922.218774:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146194560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698922.218782:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698922.218784:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698922.218786:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218788:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218791:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698922.218792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698922.218795:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218800:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698922.218805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f518738b800 x1874531146194048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698922.218808:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146194688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698922.218827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146194048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.218849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698922.218851:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698922.218852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698922.218859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787698922.218860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194048:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698922.218862:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218869:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00100000:2.0:1787698922.218873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194176:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698922.218874:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218881:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698922.218884:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698922.218895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146194816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698922.218902:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698922.218904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698922.218905:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698922.218907:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698922.218911:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6821 00000100:00080000:3.0:1787698922.218913:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698922.218916:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698922.218919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f518738bb80 x1874531146194176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698922.218921:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698922.218931:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146194944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698922.218934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146194176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.218952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.218954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.218956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698922.218961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698922.218991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698922.218999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f5187388e00 x1874531146194304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698922.219006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146194304/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.219011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.219013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.219015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698922.219018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698922.219029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698922.219035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f518738ad80 x1874531146194432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698922.219041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146194432/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.219046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.219047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.219049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146194432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698922.219059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698922.219069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698922.219075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f518738aa00 x1874531146194560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698922.219082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146194560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.219086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.219088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.219090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698922.219092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698922.219101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698922.219106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f5187389c00 x1874531146194688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698922.219112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146194688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.219116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.219118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.219119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698922.219122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698922.219130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698922.219135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f5187388380 x1874531146194816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698922.219141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146194816/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.219149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.219151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.219153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698922.219155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698922.219163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698922.219168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698922/real 1787698922] req@ffff8f5187388a80 x1874531146194944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698922.219173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146194944/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698922.219177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698922.219179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698922.219180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146194944:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698923.114531:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698924.138623:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698925.162679:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698926.186521:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698927.210510:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698927.338592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.338602:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698927.338606:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.338615:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00080000:3.0:1787698927.338626:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.338633:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698927.338665:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146195072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698927.338698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.338702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698927.338703:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.338707:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00080000:3.0:1787698927.338711:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.338715:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698927.338723:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698927.338736:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146195200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698927.338744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146195072:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698927.338750:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.338754:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698927.338756:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.338760:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00080000:3.0:1787698927.338764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.338768:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698927.338773:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698927.338787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146195328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698927.338795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.338799:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698927.338800:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.338805:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6826 00000800:00000100:2.0:1787698927.338808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698927.338810:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.338816:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698927.338821:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698927.338832:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146195456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698927.338840:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.338844:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698927.338845:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.338849:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00080000:3.0:1787698927.338853:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.338857:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698927.338863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f5187389500 x1874531146195072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698927.338904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146195072/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.338914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698927.338918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.338922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146195072:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698927.338945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698927.338969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698927.338981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f5187388000 x1874531146195200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698927.338994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146195200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698927.339005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.339009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698927.339014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698927.339033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698927.339045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f518738a300 x1874531146195328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698927.339051:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698927.339068:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146195584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698927.339069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146195328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698927.339090:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698927.339097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698927.339098:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698927.339103:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698927.339104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195328:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698927.339110:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00100000:2.0:1787698927.339114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195456:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698927.339116:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.339123:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698927.339130:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698927.339139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698927.339144:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146195712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698927.339152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f518738b100 x1874531146195456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698927.339156:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.339163:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698927.339166:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.339173:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00100000:2.0:1787698927.339175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146195456/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698927.339177:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.339185:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698927.339188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698927.339191:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698927.339194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.339198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698927.339215:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146195840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698927.339226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195584:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698927.339227:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698927.339234:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698927.339236:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.339240:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00080000:3.0:1787698927.339245:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698927.339249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698927.339252:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698927.339259:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698927.339264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f5187389180 x1874531146195584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698927.339274:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146195968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698927.339277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146195584/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698927.339302:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698927.339306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698927.339308:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787698927.339312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146195584:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787698927.339313:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698927.339322:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6826 00000100:00100000:2.0:1787698927.339324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146195712:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698927.339327:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698927.339333:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698927.339338:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698927.339348:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146196096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698927.339378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698927.339391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f518738a680 x1874531146195712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698927.339402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146195712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698927.339418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.339421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146195712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698927.339431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146195840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698927.339447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698927.339457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f5187389f80 x1874531146195840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698927.339467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146195840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698927.339477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.339479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146195840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698927.339484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146195968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698927.339498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698927.339507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f518738b480 x1874531146195968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698927.339517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146195968/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698927.339527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.339530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146195968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698927.339819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146196096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698927.339849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698927.339861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698927/real 1787698927] req@ffff8f518738b100 x1874531146196096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698927.339873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146196096/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698927.339880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698927.339883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698927.339886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146196096:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698928.234586:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698929.258615:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698930.282522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698931.307967:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698932.330674:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698932.458601:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458611:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698932.458615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458626:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458637:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458644:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698932.458679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146196224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698932.458711:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698932.458716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458721:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458725:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458729:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.458738:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698932.458752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146196352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698932.458761:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458764:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698932.458766:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458770:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458774:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458777:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.458782:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698932.458795:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146196480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698932.458798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146196224:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698932.458824:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458829:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698932.458831:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458842:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458848:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458852:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.458858:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698932.458872:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146196608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698932.458882:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458885:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698932.458887:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458891:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458894:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458899:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.458903:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698932.458915:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146196736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698932.458920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458923:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698932.458925:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458928:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458933:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458936:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.458939:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698932.458951:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146196864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698932.458955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698932.458964:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458966:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698932.458968:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458970:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.458971:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.458973:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.458975:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698932.458983:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146196992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698932.458990:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.458992:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698932.458993:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.458997:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.459001:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.459003:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.459005:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698932.459012:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146197120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698932.459017:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698932.459019:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698932.459021:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698932.459023:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6832 00000100:00080000:0.0:1787698932.459026:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698932.459029:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698932.459034:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698932.459041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace91180 x1874531146196224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698932.459049:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146197248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698932.459090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146196224/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146196224:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698932.459127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698932.459152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace90000 x1874531146196352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146196352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698932.459224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698932.459243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace90700 x1874531146196480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146196480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698932.459287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698932.459307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace93100 x1874531146196608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146196608/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698932.459350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698932.459418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace92300 x1874531146196736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146196736/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146196736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698932.459463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146196864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698932.459482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace93b80 x1874531146196864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146196864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146196864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698932.459527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146196992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698932.459561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace93800 x1874531146196992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146196992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146196992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698932.459603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146197120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698932.459622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace90380 x1874531146197120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146197120/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146197120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698932.459664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146197248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698932.459684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698932.459696:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698932/real 1787698932] req@ffff8f51ace92680 x1874531146197248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698932.459709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146197248/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698932.459717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698932.459720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698932.459723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146197248:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698933.354658:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698934.378529:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698935.402994:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698936.426558:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698937.450489:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698937.578481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578490:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698937.578494:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578504:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578514:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698937.578521:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698937.578555:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146197376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698937.578584:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698937.578589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698937.578600:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698937.578618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146197376:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698937.578622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146197504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698937.578640:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578643:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698937.578644:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578648:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578651:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698937.578654:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578659:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698937.578669:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146197632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698937.578678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698937.578681:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578685:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698937.578687:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578692:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578695:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698937.578699:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578703:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698937.578712:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146197760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698937.578741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f518738a300 x1874531146197376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698937.578768:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578772:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698937.578773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146197376/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698937.578775:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578782:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578788:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698937.578789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698937.578794:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578799:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698937.578804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.578811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146197376:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698937.578811:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146197888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698937.578825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578828:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698937.578830:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578834:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578837:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698937.578839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197504:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698937.578843:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578849:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698937.578860:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146198016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698937.578870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698937.578871:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578876:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698937.578878:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.578882:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578886:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698937.578888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f5187388000 x1874531146197504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698937.578892:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578898:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698937.578910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146197504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698937.578916:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146198144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698937.578920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.578928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.578936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197504:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698937.578941:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.578944:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698937.578945:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698937.578955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698937.578960:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.578965:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698937.578968:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698937.578973:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698937.578993:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146198272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698937.579001:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698937.579004:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698937.579006:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698937.579009:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6837 00000100:00080000:3.0:1787698937.579013:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698937.579017:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:0.0:1787698937.579018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698937.579022:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698937.579033:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146198400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698937.579045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f5187389500 x1874531146197632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146197632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698937.579078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698937.579097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698937.579108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f5187388a80 x1874531146197760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146197760/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698937.579146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698937.579165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698937.579175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f5187388380 x1874531146197888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146197888/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146197888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698937.579216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698937.579245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698937.579256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f5187389c00 x1874531146198016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146198016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698937.579289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698937.579319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698937.579330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f518738aa00 x1874531146198144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146198144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698937.579379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698937.579398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698937.579409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f518738ad80 x1874531146198272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146198272/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698937.579441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698937.579458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698937.579468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698937/real 1787698937] req@ffff8f5187388e00 x1874531146198400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698937.579478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146198400/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698937.579486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698937.579489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698937.579492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146198400:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698938.474549:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698939.498908:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698940.522514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698941.548841:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698942.570560:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698942.698463:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.698473:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698942.698476:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.698487:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.698498:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.698504:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698942.698550:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146198528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698942.698583:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.698586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698942.698588:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.698592:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.698596:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.698599:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.698608:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698942.698619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146198656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698942.698708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146198528:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698942.698712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.698716:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698942.698718:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.698723:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.698737:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.698740:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.698745:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698942.698759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146198784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698942.698769:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.698773:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698942.698774:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.698777:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.698781:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698942.698785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698942.698788:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.698794:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698942.698803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146198912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698942.698812:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.698831:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698942.698833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.698838:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.698842:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698942.698843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f518738bb80 x1874531146198528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698942.698848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.698868:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698942.698876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146198528/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698942.698880:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146199040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698942.698902:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698942.698903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698942.698907:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698942.698910:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698942.698912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698942.698916:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00100000:0.0:1787698942.698920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146198528:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698942.698932:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.698938:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.698943:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698942.698947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146198656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698942.698955:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146199168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698942.698966:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.698970:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698942.698972:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787698942.698976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698942.698979:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.698985:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.698989:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.698993:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698942.698994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f518738b800 x1874531146198656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698942.699005:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146199296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698942.699011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146198656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698942.699032:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698942.699036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146198656:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698942.699038:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698942.699041:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.699048:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.699051:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.699055:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146198784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698942.699087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f5187388e00 x1874531146198784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698942.699109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146198784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146198784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698942.699129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146198912:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787698942.699144:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698942.699149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698942.699159:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146199424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698942.699165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f518738ad80 x1874531146198912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698942.699175:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698942.699181:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698942.699184:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698942.699192:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6842 00000100:00080000:3.0:1787698942.699198:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698942.699201:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698942.699205:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698942.699211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146198912/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698942.699216:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146199552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698942.699228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146198912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698942.699247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146199040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698942.699268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f518738aa00 x1874531146199040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698942.699291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146199040/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146199040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698942.699320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698942.699338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f5187389c00 x1874531146199168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698942.699402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146199168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698942.699443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199296:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698942.699462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f5187388380 x1874531146199296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698942.699502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146199296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698942.699521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698942.699539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f5187388a80 x1874531146199424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698942.699561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146199424/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698942.699580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698942.699597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698942.699607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698942/real 1787698942] req@ffff8f5187389500 x1874531146199552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698942.699617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146199552/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698942.699624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698942.699641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698942.699644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146199552:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698943.594557:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698944.618556:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698945.645801:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698946.666598:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698947.690633:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698947.818490:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698947.818502:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818512:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818523:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818529:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698947.818591:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146199680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818621:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818624:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698947.818626:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818630:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818634:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818637:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698947.818658:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146199808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818669:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698947.818672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146199680:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698947.818673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698947.818676:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818683:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818686:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818688:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818692:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698947.818700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146199936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818706:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818708:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698947.818709:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818716:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818724:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818727:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698947.818736:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146200064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818744:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818747:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698947.818748:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818752:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818755:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818758:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818762:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698947.818773:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146200192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698947.818778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146199808:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698947.818802:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818806:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698947.818808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818813:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818818:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698947.818822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146199936:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698947.818826:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818833:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698947.818845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146200320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818853:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818856:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698947.818858:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818863:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00100000:2.0:1787698947.818863:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146200064:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698947.818869:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818874:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818878:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698947.818888:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146200448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818897:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698947.818900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146200192:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698947.818901:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698947.818904:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698947.818918:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818923:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698947.818926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200320:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698947.818929:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818936:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698947.818948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698947.818951:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146200576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698947.818969:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698947.818972:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698947.818974:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698947.818977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200576:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787698947.818981:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6847 00000100:00080000:0.0:1787698947.818987:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698947.818991:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698947.818997:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698947.819010:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146200704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698947.819953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200704:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698948.714728:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698949.738556:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698950.762608:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787698950.890804:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787698950.890859:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787698950.890876:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890920:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890932:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890948:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890955:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890963:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890969:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890976:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787698950.890983:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787698950.892587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace93800 x1874531146200704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146200704/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698950.892651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace90380 x1874531146200576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146200576/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698950.892683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace92680 x1874531146200448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146200448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892702:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698950.892715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace91500 x1874531146200192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146200192/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146200192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698950.892749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace93480 x1874531146200320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146200320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146200320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698950.892780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace91880 x1874531146199808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146199808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146199808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698950.892809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace92d80 x1874531146199936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146199936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146199936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698950.892837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace90e00 x1874531146200064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146200064/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146200064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698950.892864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698947/real 1787698950] req@ffff8f51ace91c00 x1874531146199680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698950.892875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146199680/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698950.892882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698950.892885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698950.892888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146199680:192.168.201.125@tcp:250:kworker.0 00010000:00010000:1.0:1787698951.786643:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698952.810561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698952.938524:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.938538:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698952.938542:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938552:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:3.0:1787698952.938561:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938568:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698952.938610:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146200832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698952.938637:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.938641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698952.938643:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938647:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:3.0:1787698952.938651:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938655:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698952.938665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698952.938677:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146200960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698952.938688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146200832:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698952.938698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.938702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698952.938704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938708:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:3.0:1787698952.938712:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938715:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698952.938720:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698952.938733:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146201088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698952.938742:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.938745:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698952.938747:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938750:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6852 00000800:00000100:2.0:1787698952.938755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698952.938756:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938763:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698952.938768:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698952.938779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146201216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698952.938788:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.938791:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698952.938793:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938798:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:3.0:1787698952.938802:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938807:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698952.938811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f5187388000 x1874531146200832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698952.938813:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698952.938826:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146201344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698952.938832:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698952.938837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146200832/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698952.938837:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698952.938842:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938849:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:2.0:1787698952.938854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698952.938855:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938862:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698952.938864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698952.938869:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698952.938871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146200832:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698952.938885:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146201472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698952.938896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.938900:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787698952.938901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146200960:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698952.938902:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938910:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:3.0:1787698952.938915:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.938919:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698952.938924:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787698952.938931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698952.938936:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146201600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698952.938948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f518738a300 x1874531146200960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698952.938968:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698952.938969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146200960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698952.938975:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698952.938978:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698952.938985:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:2.0:1787698952.938986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698952.938990:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698952.938994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698952.938997:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146200960:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698952.939004:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698952.939014:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146201088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698952.939020:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146201728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698952.939030:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698952.939034:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698952.939036:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698952.939040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698952.939043:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6852 00000100:00080000:3.0:1787698952.939049:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698952.939053:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f518738b100 x1874531146201088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698952.939060:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698952.939076:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146201856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698952.939077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146201088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146201088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698952.939138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146201216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698952.939158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f518738b480 x1874531146201216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.939180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146201216/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146201216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698952.939198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146201344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698952.939216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f5187389f80 x1874531146201344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.939236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146201344/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146201344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698952.939256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698952.939275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f518738a680 x1874531146201472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.939296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146201472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698952.939328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698952.939347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f5187389180 x1874531146201600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.939399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146201600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698952.939418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698952.939444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f5187388000 x1874531146201728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.939479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146201728/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698952.939498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698952.939516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698952.939526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff8f5187389500 x1874531146201856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.939537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146201856/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.939544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.939547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698952.939550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146201856:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698953.834607:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698954.858534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698955.882578:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698956.906565:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698957.930589:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698958.058491:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058502:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698958.058506:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058516:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058527:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058541:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698958.058578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146201984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058609:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058612:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698958.058614:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058618:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058621:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058632:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058640:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698958.058652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146202112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058660:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058663:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698958.058664:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058667:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058670:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058673:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058677:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698958.058686:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146202240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058695:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698958.058699:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058702:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058705:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058708:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698958.058718:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146202368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058727:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698958.058728:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058731:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058733:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058736:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698958.058748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146202496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698958.058760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058763:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058765:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058768:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058772:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698958.058778:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146202624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058785:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058798:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698958.058799:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058802:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058805:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058808:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058812:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698958.058821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146202752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058828:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058830:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698958.058832:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058835:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058838:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058841:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698958.058851:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146202880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698958.058857:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698958.058860:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698958.058861:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698958.058864:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6857 00000100:00080000:0.0:1787698958.058866:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698958.058869:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698958.058872:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698958.058879:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146203008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698958.058961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146201984:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787698958.059036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace93b80 x1874531146201984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146201984/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146201984:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698958.059124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698958.059144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace92300 x1874531146202112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146202112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698958.059182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698958.059199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace93100 x1874531146202240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146202240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698958.059236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698958.059269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace90700 x1874531146202368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146202368/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698958.059306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698958.059320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace90000 x1874531146202496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146202496/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146202496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698958.059344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146202624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698958.059415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace91180 x1874531146202624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146202624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146202624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698958.059447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146202752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698958.059459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace91c00 x1874531146202752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146202752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146202752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698958.059480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146202880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698958.059489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace90e00 x1874531146202880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146202880/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146202880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698958.059514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146203008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698958.059527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698958.059536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698958/real 1787698958] req@ffff8f51ace92d80 x1874531146203008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698958.059546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146203008/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698958.059553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698958.059556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698958.059558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146203008:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698958.954520:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698959.978553:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698961.002464:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698962.026535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698963.050536:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698963.178497:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698963.178507:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698963.178511:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698963.178521:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:3.0:1787698963.178534:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698963.178540:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698963.178576:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146203136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698963.178601:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698963.178605:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787698963.178650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146203136:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698963.178697:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698963.178710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698963.178711:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:3.0:1787698963.178718:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698963.178723:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698963.178740:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698963.178754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f5187388a80 x1874531146203136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698963.178759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146203264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698963.178780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698963.178784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146203136/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.178796:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698963.178800:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698963.178802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698963.178806:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:2.0:1787698963.178809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698963.178813:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698963.178816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146203136:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698963.178820:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698963.178834:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698963.178843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698963.178850:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146203392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698963.178861:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698963.178864:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698963.178866:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698963.178871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698963.178875:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:3.0:1787698963.178881:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698963.178885:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698963.178889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f5187388380 x1874531146203264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698963.178892:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698963.178902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146203264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698963.178910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698963.178914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.178916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698963.178923:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146203520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698963.178935:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698963.178936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203392:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698963.178940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698963.178943:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698963.178949:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:3.0:1787698963.178953:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698963.178957:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698963.178962:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698963.178967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698963.178977:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146203648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698963.178983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f5187389c00 x1874531146203392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.178992:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698963.178997:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698963.178999:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698963.179001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146203392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.179005:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:3.0:1787698963.179011:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698963.179014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698963.179017:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698963.179020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.179025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203392:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698963.179026:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698963.179035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698963.179041:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146203776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698963.179052:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698963.179054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698963.179058:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698963.179061:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698963.179065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f518738aa00 x1874531146203520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.179066:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00080000:3.0:1787698963.179073:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698963.179079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146203520/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.179080:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698963.179090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698963.179090:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698963.179095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.179098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698963.179102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787698963.179107:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146203904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698963.179117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698963.179117:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698963.179123:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698963.179125:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698963.179127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f518738ad80 x1874531146203648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.179133:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00100000:2.0:1787698963.179139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146203648/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.179139:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698963.179147:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787698963.179149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698963.179154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787698963.179156:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698963.179157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146203648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698963.179170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146203776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698963.179174:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146204032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698963.179184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698963.179187:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698963.179191:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787698963.179194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f5187388e00 x1874531146203776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698963.179195:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698963.179203:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6862 00000100:00100000:2.0:1787698963.179207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146203776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698963.179218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698963.179219:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698963.179221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.179225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146203776:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698963.179228:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698963.179232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146203904:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787698963.179238:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698963.179244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698963.179257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f518738b800 x1874531146203904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698963.179257:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146204160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698963.179273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146203904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698963.179312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698963.179317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.179321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146203904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698963.179333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146204032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698963.179350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698963.179378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f518738bb80 x1874531146204032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698963.179385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146204032/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698963.179390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698963.179392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.179394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146204032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698963.179398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146204160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698963.179408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698963.179414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698963/real 1787698963] req@ffff8f518738ad80 x1874531146204160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698963.179420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146204160/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698963.179425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698963.179426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698963.179427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146204160:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698964.074524:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698965.098550:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698966.122569:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698967.146568:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698968.170563:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698968.298443:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298454:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698968.298459:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298470:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298492:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698968.298502:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698968.298541:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146204288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698968.298569:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298573:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698968.298575:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298588:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298595:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698968.298600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298610:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698968.298613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146204288:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698968.298627:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146204416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698968.298647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298651:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698968.298652:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298657:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298661:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698968.298664:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298669:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698968.298681:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146204544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698968.298694:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298697:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698968.298699:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298703:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298707:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698968.298712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698968.298719:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298727:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698968.298744:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146204672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698968.298755:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698968.298760:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298766:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298772:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698968.298774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f518738aa00 x1874531146204288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698968.298780:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298786:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698968.298806:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146204800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698968.298817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146204288/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698968.298818:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298829:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698968.298833:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298843:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298846:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698968.298850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698968.298851:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298855:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698968.298862:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146204928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698968.298870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698968.298871:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298877:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698968.298881:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698968.298882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146204288:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698968.298889:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298894:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698968.298899:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698968.298915:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146205056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698968.298933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204416:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698968.298934:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298946:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698968.298949:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698968.298953:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.298958:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698968.298961:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.298965:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698968.298975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698968.298977:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146205184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698968.298986:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698968.298989:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698968.298991:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698968.298996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187389c00 x1874531146204416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698968.298996:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6867 00000100:00080000:3.0:1787698968.299003:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698968.299011:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698968.299017:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698968.299033:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146205312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698968.299039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146204416/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698968.299090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204544:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698968.299108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187388380 x1874531146204544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146204544/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204544:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698968.299144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698968.299165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187388a80 x1874531146204672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146204672/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204672:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698968.299228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698968.299243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187389500 x1874531146204800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146204800/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146204800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698968.299270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146204928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698968.299283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187388000 x1874531146204928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146204928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146204928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698968.299338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146205056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698968.299379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187389180 x1874531146205056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146205056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146205056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698968.299411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146205184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698968.299423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f518738a680 x1874531146205184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146205184/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146205184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698968.299447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146205312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698968.299456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698968.299462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698968/real 1787698968] req@ffff8f5187389f80 x1874531146205312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698968.299469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146205312/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698968.299474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698968.299475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698968.299477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146205312:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698969.194555:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698970.218631:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698971.242645:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698972.266583:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698973.290622:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698973.418483:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418496:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698973.418506:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418519:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418534:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418542:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698973.418587:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146205440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698973.418614:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698973.418618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418622:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418625:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418627:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418638:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698973.418651:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146205568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698973.418662:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698973.418663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146205440:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698973.418668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698973.418672:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418677:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418681:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418683:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418688:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698973.418699:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146205696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698973.418706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698973.418710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418712:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418722:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418725:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698973.418735:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146205824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698973.418743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698973.418745:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418748:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698973.418750:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418752:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418755:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418757:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418760:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698973.418767:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146205952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698973.418773:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698973.418777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418782:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418785:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418789:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698973.418795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f518738b480 x1874531146205440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698973.418798:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146206080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698973.418818:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418821:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698973.418822:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698973.418827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146205440/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698973.418827:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418828:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418834:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418839:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787698973.418845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698973.418849:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146206208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698973.418852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.418859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146205440:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698973.418862:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418865:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698973.418866:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418870:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418872:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418875:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418877:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698973.418882:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146206336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698973.418887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205568:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698973.418891:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698973.418893:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698973.418895:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698973.418898:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6873 00000100:00080000:3.0:1787698973.418901:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698973.418903:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698973.418906:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698973.418910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698973.418916:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146206464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698973.418923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f518738b100 x1874531146205568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.418933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146205568/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.418943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.418945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.418948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698973.418952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205696:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698973.418981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.418992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f518738a300 x1874531146205696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146205696/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698973.419023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698973.419043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.419054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f5187389f80 x1874531146205824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146205824/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205824:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698973.419084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698973.419098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.419131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f518738a680 x1874531146205952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146205952/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146205952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698973.419151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698973.419165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.419174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f5187389180 x1874531146206080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146206080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698973.419205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698973.419224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.419236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f5187388000 x1874531146206208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146206208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698973.419290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698973.419309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.419319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f5187389500 x1874531146206336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146206336/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698973.419339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698973.419382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698973.419393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698973/real 1787698973] req@ffff8f5187388a80 x1874531146206464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698973.419401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146206464/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698973.419407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698973.419409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698973.419411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146206464:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698974.314525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698975.338630:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698976.362530:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698977.386626:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698978.410591:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698978.538586:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.538600:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698978.538605:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.538618:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.538631:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698978.538639:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698978.538689:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146206592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698978.538726:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.538730:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698978.538732:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.538737:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.538741:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698978.538745:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.538757:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698978.538771:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146206720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698978.538783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146206592:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698978.538783:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.538792:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698978.538795:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.538800:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.538803:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698978.538807:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.538812:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698978.538823:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146206848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698978.538832:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.538836:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698978.538837:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.538841:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.538844:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698978.538848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.538854:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698978.538864:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146206976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698978.538869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698978.538879:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.538882:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698978.538884:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.538890:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.538894:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698978.538898:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.538903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698978.538923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704df80 x1874531146206592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698978.538927:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146207104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698978.538955:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.538959:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698978.538961:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698978.538966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146206592/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698978.538967:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.538972:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698978.538979:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.538984:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787698978.538990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.538997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698978.538999:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146207232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698978.539005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146206592:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787698978.539017:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.539035:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698978.539037:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.539042:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.539046:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698978.539050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146206720:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787698978.539051:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.539060:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698978.539072:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146207360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698978.539080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698978.539090:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698978.539095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704c700 x1874531146206720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698978.539101:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698978.539105:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698978.539110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146206720/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698978.539113:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.539121:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698978.539122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787698978.539129:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.539139:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698978.539142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146206720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698978.539148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146206848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698978.539154:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146207488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698978.539164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787698978.539166:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698978.539171:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698978.539174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704fb80 x1874531146206848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698978.539174:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698978.539183:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6878 00000100:00080000:2.0:1787698978.539190:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698978.539197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146206848/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698978.539197:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787698978.539206:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698978.539207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146206848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698978.539219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146206976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698978.539237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698978.539239:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146207616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698978.539249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704d180 x1874531146206976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698978.539262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146206976/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698978.539269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146206976:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698978.539280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146207104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698978.539298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698978.539307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704c000 x1874531146207104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698978.539316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146207104/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698978.539328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146207104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698978.539338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698978.539394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698978.539406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704c380 x1874531146207232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698978.539417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146207232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698978.539423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698978.539441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698978.539460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698978.539466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704ce00 x1874531146207360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698978.539473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146207360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698978.539478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698978.539486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698978.539495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698978.539500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704f800 x1874531146207488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698978.539506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146207488/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698978.539510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698978.539516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698978.539526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698978.539535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698978/real 1787698978] req@ffff8f518704df80 x1874531146207616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698978.539545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146207616/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698978.539552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698978.539555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698978.539558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146207616:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698979.434664:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698980.458595:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698981.482575:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698982.506639:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698983.530541:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698983.658552:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698983.658565:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698983.658570:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.658583:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.658597:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.658604:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698983.658649:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146207744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.658679:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698983.658682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698983.658684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.658689:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.658692:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.658696:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.658708:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698983.658720:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146207872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.658736:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698983.658739:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698983.658741:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.658745:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.658749:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.658752:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.658756:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698983.658768:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146208000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.658782:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698983.658783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146207744:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698983.658792:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698983.658797:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.658802:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.658805:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.658809:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.658814:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698983.658827:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146208128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.658836:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698983.658840:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698983.658843:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.658849:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.658853:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.658857:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.658862:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698983.658873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787698983.658876:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146208256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.658888:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698983.658892:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698983.658894:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.658900:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.658903:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.658910:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.658915:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698983.658930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f5187388380 x1874531146207744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698983.658965:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146208384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698983.658976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146207744/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698983.658983:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698983.658997:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698983.659003:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698983.659008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698983.659012:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:0.0:1787698983.659018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698983.659020:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.659030:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146207744:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787698983.659037:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698983.659046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146207872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787698983.659054:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146208512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.659071:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698983.659078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787698983.659086:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698983.659089:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698983.659097:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00100000:0.0:1787698983.659098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f5187389c00 x1874531146207872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698983.659104:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.659113:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.659124:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698983.659128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146207872/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698983.659144:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146208640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698983.659164:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698983.659167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698983.659169:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698983.659175:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698983.659180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146207872:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787698983.659184:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6883 00000100:00080000:3.0:1787698983.659193:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698983.659200:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698983.659205:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698983.659217:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146208768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698983.659224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146208000:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698983.659259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f518738aa00 x1874531146208000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146208000/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146208000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698983.659316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146208128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698983.659335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f518738ad80 x1874531146208128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146208128/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146208128:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698983.659483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146208256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698983.659504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f518738bb80 x1874531146208256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146208256/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146208256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698983.659564:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698983.659582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f518738b800 x1874531146208384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146208384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698983.659637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787698983.659656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f5187388e00 x1874531146208512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146208512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659695:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698983.659708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698983.659726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f5187388a80 x1874531146208640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146208640/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787698983.659779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787698983.659797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787698983.659807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698983/real 1787698983] req@ffff8f5187389500 x1874531146208768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698983.659817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146208768/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698983.659825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698983.659828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787698983.659831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146208768:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698984.554607:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698985.578572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698986.602678:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698987.626634:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698988.650605:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698988.778481:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698988.778496:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698988.778501:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778513:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778526:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778534:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698988.778581:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146208896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698988.778613:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698988.778617:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698988.778619:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778623:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778626:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778631:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778642:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698988.778655:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146209024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698988.778667:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698988.778672:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787698988.778673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146208896:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698988.778674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778683:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778687:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778691:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778696:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698988.778710:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146209152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698988.778720:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698988.778724:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698988.778725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778729:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778732:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778736:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778741:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698988.778752:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146209280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698988.778762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698988.778766:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698988.778771:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698988.778774:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778779:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778783:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778787:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778792:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698988.778803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146209408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698988.778813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698988.778816:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698988.778818:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698988.778822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace91880 x1874531146208896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698988.778826:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778832:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778839:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778859:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698988.778864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146208896/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698988.778875:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146209536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698988.778876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.778886:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787698988.778891:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698988.778895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146208896:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787698988.778897:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698988.778901:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778906:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.778910:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778915:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778919:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698988.778925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787698988.778931:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146209664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698988.778944:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698988.778949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698988.778951:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698988.778954:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.778959:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00100000:2.0:1787698988.778963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace93480 x1874531146209024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698988.778965:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698988.778973:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698988.778978:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698988.778986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146209024/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698988.778992:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146209792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698988.778994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209024:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698988.779015:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698988.779016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209152:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787698988.779024:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698988.779027:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698988.779032:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6888 00000100:00080000:0.0:1787698988.779037:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698988.779037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787698988.779044:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace91500 x1874531146209152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698988.779051:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698988.779061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146209152/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787698988.779069:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146209920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698988.779072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698988.779087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698988.779104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace92680 x1874531146209280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698988.779127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146209280/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209280:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698988.779145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698988.779164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace90380 x1874531146209408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698988.779187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146209408/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146209408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698988.779206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698988.779224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace93800 x1874531146209536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698988.779239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146209536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698988.779260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698988.779271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace91880 x1874531146209664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698988.779282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146209664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698988.779293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698988.779302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace92d80 x1874531146209792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698988.779313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146209792/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698988.779333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698988.779347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698988.779380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698988/real 1787698988] req@ffff8f51ace90e00 x1874531146209920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698988.779392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146209920/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698988.779400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698988.779403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698988.779406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146209920:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698989.674979:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698990.698653:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698991.722546:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698992.749504:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698993.770534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698993.898496:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898512:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698993.898517:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898531:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898548:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.898557:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787698993.898608:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146210048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898640:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898645:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698993.898646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898652:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898655:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.898660:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.898672:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698993.898696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146210048:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787698993.898700:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146210176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898713:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898716:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698993.898718:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898722:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898726:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.898731:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.898735:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698993.898750:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146210304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898775:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898780:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698993.898781:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898785:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898788:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.898791:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.898795:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698993.898804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146210432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898809:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898819:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698993.898821:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898824:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898828:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.898832:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.898837:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698993.898848:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146210560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898857:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898861:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698993.898863:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898867:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898871:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.898876:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787698993.898878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787698993.898883:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787698993.898910:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146210688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898920:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.898924:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698993.898926:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.898930:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.898934:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698993.898938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f5187388000 x1874531146210048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698993.898941:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.898949:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698993.898964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146210048/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698993.898965:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146210816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.898994:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698993.898996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698993.898998:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698993.899002:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698993.899003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787698993.899010:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00100000:2.0:1787698993.899013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146210048:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787698993.899016:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.899022:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.899027:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698993.899037:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146210944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787698993.899045:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698993.899048:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698993.899050:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698993.899053:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6893 00000100:00080000:3.0:1787698993.899057:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787698993.899060:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787698993.899065:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787698993.899074:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146211072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698993.899555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698993.899589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f5187389180 x1874531146210176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146210176/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698993.899629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210304:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698993.899643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f518738a680 x1874531146210304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146210304/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210304:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698993.899670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698993.899683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f5187389f80 x1874531146210432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146210432/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210432:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698993.899724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698993.899739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f518738a300 x1874531146210560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146210560/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146210560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698993.899781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146210688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698993.899795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f518738b100 x1874531146210688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146210688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899815:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146210688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698993.899823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146210816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698993.899832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f518738b480 x1874531146210816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146210816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146210816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698993.899866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146210944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698993.899876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f5187389500 x1874531146210944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146210944/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146210944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698993.899898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146211072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698993.899907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698993.899913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff8f5187388a80 x1874531146211072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698993.899919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146211072/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698993.899924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698993.899925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698993.899927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146211072:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787698994.794584:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698995.818594:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787698996.842609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698997.866611:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787698998.890525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698999.018469:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018479:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698999.018484:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018493:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018505:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018512:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787698999.018557:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146211200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018590:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018594:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698999.018595:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018599:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018602:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018605:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018617:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698999.018630:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146211328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018640:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698999.018646:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018650:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018654:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018657:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018661:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698999.018669:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146211456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018689:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698999.018693:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018700:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018709:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698999.018720:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146211584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018728:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018731:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698999.018733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018737:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018740:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018744:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018748:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698999.018758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146211712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018767:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018771:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698999.018773:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018777:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018780:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018785:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018789:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698999.018799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146211840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018808:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018811:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698999.018814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018817:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018821:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018824:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018829:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698999.018839:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146211968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018846:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018848:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698999.018850:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018852:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018855:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018857:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018860:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698999.018870:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146212096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698999.018877:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698999.018879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698999.018880:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698999.018883:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6898 00000100:00080000:0.0:1787698999.018884:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698999.018886:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787698999.018889:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698999.018896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146212224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698999.018964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146211200:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787698999.019032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace91c00 x1874531146211200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146211200/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146211200:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787698999.019143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211328:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698999.019167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace91180 x1874531146211328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146211328/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698999.019219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698999.019239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace90000 x1874531146211456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146211456/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698999.019282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211584:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698999.019302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace90700 x1874531146211584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146211584/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698999.019345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698999.019392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace93100 x1874531146211712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146211712/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146211712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698999.019421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146211840:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698999.019433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace92300 x1874531146211840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146211840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146211840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698999.019468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146211968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787698999.019487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace93b80 x1874531146211968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146211968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146211968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787698999.019529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146212096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698999.019549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace90e00 x1874531146212096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146212096/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146212096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787698999.019591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146212224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787698999.019610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787698999.019622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698999/real 1787698999] req@ffff8f51ace92d80 x1874531146212224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698999.019633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146212224/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698999.019641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698999.019644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787698999.019647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146212224:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787698999.914596:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699000.938574:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699001.962609:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699002.986564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699004.010546:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699004.138450:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699004.138460:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699004.138472:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699004.138480:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138490:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699004.138494:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699004.138521:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146212352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699004.138541:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699004.138545:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699004.138547:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699004.138550:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138553:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699004.138557:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138572:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699004.138575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146212352:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699004.138595:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146212480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699004.138605:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699004.138608:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699004.138610:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699004.138613:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138616:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699004.138619:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138623:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699004.138631:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146212608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699004.138638:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699004.138640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699004.138641:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699004.138643:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699004.138645:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138648:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699004.138651:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138654:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699004.138662:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146212736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699004.138669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699004.138671:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699004.138673:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699004.138675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704e680 x1874531146212352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699004.138676:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138679:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699004.138682:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138685:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699004.138690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146212352/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699004.138693:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146212864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699004.138700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699004.138701:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699004.138703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787699004.138704:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699004.138706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699004.138708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146212352:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787699004.138709:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138712:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699004.138715:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138718:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699004.138726:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146212992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699004.138727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699004.138735:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699004.138737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699004.138739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699004.138742:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138745:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699004.138746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699004.138748:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138751:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699004.138756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704f480 x1874531146212480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699004.138759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146213120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699004.138767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146212480/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699004.138768:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699004.138771:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699004.138772:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699004.138775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699004.138776:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:0.0:1787699004.138778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787699004.138779:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699004.138781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212480:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699004.138781:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138785:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699004.138789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699004.138792:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146213248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699004.138800:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699004.138803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699004.138803:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699004.138805:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699004.138808:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6903 00000100:00080000:2.0:1787699004.138811:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699004.138812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704ed80 x1874531146212608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699004.138815:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699004.138818:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699004.138822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146212608/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699004.138828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146213376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699004.138829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.138832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.138835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212608:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699004.138840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699004.138857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699004.138868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704dc00 x1874531146212736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699004.138878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146212736/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699004.138884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.138885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.138887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212736:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699004.138892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699004.138909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699004.138919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704ea00 x1874531146212864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699004.138931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146212864/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699004.138939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.138941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.138944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146212864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699004.138957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146212992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699004.138971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699004.138977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704e300 x1874531146212992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699004.138987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146212992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699004.138995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.138997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.139000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146212992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699004.139011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146213120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699004.139029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699004.139039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704d880 x1874531146213120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699004.139049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146213120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699004.139062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.139065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.139068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146213120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699004.139073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146213248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699004.139090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699004.139100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704d500 x1874531146213248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699004.139109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146213248/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699004.139117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.139119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.139122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146213248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699004.139127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146213376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699004.139143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699004.139152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699004/real 1787699004] req@ffff8f518704e680 x1874531146213376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699004.139163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146213376/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699004.139170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699004.139172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699004.139176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146213376:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699005.034617:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699006.058561:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699007.082583:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699008.106557:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699009.130579:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699009.258451:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258460:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699009.258464:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258474:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:2.0:1787699009.258484:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699009.258491:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699009.258549:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146213504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699009.258576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258580:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699009.258582:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258586:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:2.0:1787699009.258590:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699009.258593:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258603:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699009.258619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146213632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699009.258622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146213504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787699009.258637:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258640:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699009.258642:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258647:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:2.0:1787699009.258650:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699009.258654:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258660:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699009.258671:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146213760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699009.258680:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258684:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699009.258686:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258691:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6908 00000800:00000100:0.0:1787699009.258693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699009.258695:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699009.258700:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258705:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699009.258715:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146213888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699009.258725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258728:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699009.258730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258734:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00100000:0.0:1787699009.258739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704df80 x1874531146213504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699009.258739:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699009.258746:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258754:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699009.258765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146213504/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699009.258767:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146214016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699009.258793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699009.258793:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699009.258803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787699009.258804:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258808:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:2.0:1787699009.258813:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699009.258817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146213504:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787699009.258821:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258827:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699009.258839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146214144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699009.258845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146213632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699009.258855:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258859:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699009.258862:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699009.258867:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:2.0:1787699009.258871:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699009.258876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699009.258877:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258884:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699009.258892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704f800 x1874531146213632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699009.258898:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146214272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699009.258914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146213632/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699009.258914:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258922:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699009.258927:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699009.258932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699009.258934:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:0.0:1787699009.258939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787699009.258941:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699009.258945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146213632:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699009.258948:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.258956:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699009.258959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146213760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699009.258971:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146214400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699009.258981:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699009.258984:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699009.258987:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787699009.258988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699009.258994:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6908 00000100:00080000:2.0:1787699009.259000:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699009.259005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704ce00 x1874531146213760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699009.259005:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699009.259012:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699009.259025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146213760/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699009.259028:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146214528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699009.259045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146213760:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699009.259061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146213888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699009.259082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699009.259092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704c380 x1874531146213888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699009.259104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146213888/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699009.259112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146213888:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699009.259126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146214016:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699009.259144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699009.259155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704c000 x1874531146214016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699009.259173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146214016/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699009.259181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146214016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699009.259204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699009.259224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699009.259235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704d180 x1874531146214144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699009.259246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146214144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699009.259254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699009.259265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699009.259284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699009.259303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704fb80 x1874531146214272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699009.259314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146214272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699009.259328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699009.259340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699009.259411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699009.259424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704c700 x1874531146214400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699009.259435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146214400/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699009.259443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699009.259454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699009.259472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699009.259482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699009/real 1787699009] req@ffff8f518704df80 x1874531146214528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699009.259493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146214528/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699009.259515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699009.259518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699009.259522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146214528:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699010.154573:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699011.178547:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699012.202557:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699013.226553:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699014.250544:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699014.378567:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378581:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699014.378586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.378600:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378614:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.378624:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699014.378666:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146214656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.378698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699014.378704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.378709:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378713:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.378718:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.378729:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699014.378743:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146214784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.378756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699014.378760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699014.378765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146214656:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699014.378766:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378770:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.378776:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.378780:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699014.378790:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146214912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.378798:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378801:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699014.378803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.378807:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378810:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.378814:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.378819:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699014.378830:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146215040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.378839:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378843:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699014.378844:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.378847:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378851:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.378853:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.378857:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699014.378868:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146215168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.378876:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378879:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699014.378881:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.378886:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378889:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.378894:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.378900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699014.378912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146214784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699014.378916:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146215296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.378929:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.378933:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699014.378935:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.378939:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.378943:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699014.378950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146214912:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699014.378951:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.378962:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699014.378979:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146215424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699014.378995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146215040:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699014.378995:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.379003:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699014.379006:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.379011:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.379015:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.379020:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.379026:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699014.379027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146215168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699014.379044:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146215552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699014.379055:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699014.379059:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787699014.379062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215296:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699014.379063:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699014.379072:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6914 00000100:00080000:3.0:1787699014.379076:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699014.379080:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699014.379084:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699014.379094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699014.379096:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146215680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699014.379123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699014.379145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215680:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699015.274615:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699016.298633:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699017.322620:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787699017.450772:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787699017.450842:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787699017.450863:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450930:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450952:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450963:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450971:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450980:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450989:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.450997:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699017.451005:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787699017.451268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f5187389500 x1874531146215680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146215680/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.451328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f5187389c00 x1874531146215296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146215296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699017.451473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f5187388380 x1874531146215424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146215424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699017.451514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f5187388a80 x1874531146215552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146215552/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146215552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.451548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f518738b800 x1874531146214784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146214784/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146214784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699017.451582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f518738bb80 x1874531146214912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146214912/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146214912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699017.451616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f518738ad80 x1874531146215040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146215040/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146215040:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.451649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f518738aa00 x1874531146215168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146215168/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146215168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699017.451682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699014/real 1787699017] req@ffff8f5187388e00 x1874531146214656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699017.451694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146214656/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699017.451702:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699017.451705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699017.451709:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146214656:192.168.201.125@tcp:250:kworker.0 00010000:00010000:0.0:1787699018.346571:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699019.370593:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699019.498662:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.498673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699019.498677:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.498687:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.498701:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.498709:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699019.498756:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146215808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.498795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.498800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699019.498806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.498811:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.498815:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.498819:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.498831:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699019.498845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146215936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.498855:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.498859:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699019.498861:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.498865:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.498868:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.498872:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.498877:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699019.498890:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146216064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.498899:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.498902:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699019.498904:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.498908:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.498911:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.498915:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.498920:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699019.498930:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146216192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.498939:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.498943:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699019.498944:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.498949:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.498952:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.498955:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.498959:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699019.498970:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146216320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.498977:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.498981:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699019.498983:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.498988:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.498991:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.498995:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.499000:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699019.499010:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146216448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.499019:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.499022:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699019.499024:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.499028:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.499031:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.499035:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.499039:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699019.499049:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146216576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.499057:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.499061:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699019.499063:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.499066:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.499069:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.499072:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.499077:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699019.499089:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146216704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699019.499098:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699019.499102:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699019.499104:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699019.499108:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6919 00000100:00080000:3.0:1787699019.499111:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699019.499115:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699019.499119:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699019.499130:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146216832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699019.499256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146215808:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787699019.499438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.499487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f518738b480 x1874531146215808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.499509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146215808/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.499522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.499527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.499531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146215808:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699019.499538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146215936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699019.499563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.499576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f518738b100 x1874531146215936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.499588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146215936/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.499596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.499599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.499603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146215936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699019.499607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146216064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699019.499627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.499637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f518738a300 x1874531146216064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.499647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146216064/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.499655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.499657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.499660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146216064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699019.499667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146216192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699019.499685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.499696:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f5187389f80 x1874531146216192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.499707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146216192/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.499714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.499717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.499720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146216192:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699019.499726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146216320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699019.499744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.499753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f518738a680 x1874531146216320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.499763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146216320/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.499770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.499772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.499775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146216320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699019.500949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699019.500994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.501011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f5187389180 x1874531146216448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.501188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146216448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.501199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.501204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.501208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699019.501215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699019.501235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.501243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f5187388000 x1874531146216576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.501251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146216576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.501256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.501260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.501270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699019.501277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699019.501297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.501308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f5187388e00 x1874531146216704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.501319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146216704/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.501347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.501350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.501373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699019.501380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699019.501405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699019.501416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699019/real 1787699019] req@ffff8f518738aa00 x1874531146216832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699019.501427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146216832/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699019.501435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699019.501438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699019.501441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146216832:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699020.394517:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699021.418539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699022.442665:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699023.466541:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699024.490554:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699024.618476:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618484:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699024.618487:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618495:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618503:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618509:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699024.618537:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146216960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618556:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699024.618560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618565:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618567:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618573:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699024.618580:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146217088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618588:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699024.618589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618591:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618594:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618598:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699024.618615:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146217216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618624:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618627:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699024.618633:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618637:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618641:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699024.618650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146216960:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699024.618651:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618662:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699024.618679:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146217344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618689:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618693:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699024.618694:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618698:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618701:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618704:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618709:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699024.618719:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146217472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618728:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699024.618730:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618732:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618734:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618737:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618740:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699024.618746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699024.618748:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146217600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618756:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618759:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699024.618762:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618765:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618768:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618772:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618775:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699024.618786:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146217728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699024.618791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618793:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699024.618795:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618797:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618799:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699024.618802:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618807:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699024.618816:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146217856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699024.618823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183647b80 x1874531146216960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699024.618825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699024.618840:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699024.618843:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699024.618860:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6924 00000100:00080000:1.0:1787699024.618867:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699024.618875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146216960/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699024.618875:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699024.618891:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787699024.618896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.618905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699024.618909:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146217984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699024.618914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146216960:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699024.618939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699024.618975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.618988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183646a00 x1874531146217088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146217088/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699024.619024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217216:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699024.619041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183646d80 x1874531146217216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146217216/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699024.619102:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699024.619123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183644e00 x1874531146217344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146217344/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217344:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699024.619168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217472:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699024.619183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183644700 x1874531146217472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146217472/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146217472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699024.619235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699024.619263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183647800 x1874531146217600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146217600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699024.619310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699024.619327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183646680 x1874531146217728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146217728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699024.619403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699024.619421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183644000 x1874531146217856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146217856/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699024.619464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699024.619480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699024.619488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699024/real 1787699024] req@ffff8f5183645c00 x1874531146217984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699024.619498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146217984/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699024.619505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699024.619508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699024.619511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146217984:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699025.514565:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699026.538569:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699027.562564:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699028.586485:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699029.610536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699029.738493:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699029.738500:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699029.738502:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699029.738510:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738518:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738529:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699029.738557:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146218112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699029.738576:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699029.738579:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699029.738580:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699029.738584:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699029.738605:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699029.738619:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146218240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699029.738636:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699029.738638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146218112:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699029.738639:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699029.738641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699029.738647:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738650:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738652:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699029.738655:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699029.738665:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146218368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699029.738671:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699029.738673:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699029.738674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699029.738676:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738679:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738680:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699029.738683:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699029.738691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146218496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699029.738697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699029.738702:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699029.738705:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699029.738706:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699029.738709:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738712:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738715:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699029.738717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699029.738724:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146218624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699029.738730:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699029.738732:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699029.738733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699029.738736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f518738ad80 x1874531146218112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699029.738745:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738748:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738750:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699029.738754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146218112/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699029.738755:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699029.738766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699029.738767:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146218752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699029.738774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699029.738780:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699029.738781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146218112:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699029.738786:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699029.738789:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699029.738792:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738793:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738796:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699029.738798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699029.738804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146218880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699029.738807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218240:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699029.738826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699029.738830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699029.738833:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699029.738836:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699029.738843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f518738bb80 x1874531146218240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699029.738844:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:3.0:1787699029.738850:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699029.738856:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699029.738864:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699029.738867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146218240/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699029.738875:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146219008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699029.738882:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699029.738885:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699029.738889:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699029.738891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699029.738893:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6929 00000100:00080000:0.0:1787699029.738899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699029.738900:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699029.738905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218240:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699029.738909:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699029.738914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218368:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699029.738916:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699029.738930:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146219136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699029.738937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.738951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f518738b800 x1874531146218368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.738959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146218368/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.738964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.738966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.738969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699029.738972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699029.738983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.738989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f5187388a80 x1874531146218496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.738995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146218496/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.738999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.739001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.739002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218496:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699029.739006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699029.739014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.739019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f5187388380 x1874531146218624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.739025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146218624/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.739029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.739030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.739032:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146218624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699029.739034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146218752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699029.739042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.739047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f5187389c00 x1874531146218752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.739057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146218752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.739062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.739063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.739065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146218752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699029.739076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146218880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699029.739085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.739092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f5187389500 x1874531146218880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.739102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146218880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.739109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.739112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.739115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146218880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699029.739120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146219008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699029.739137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.739144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f518738ad80 x1874531146219008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.739174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146219008/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.739183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.739185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.739186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146219008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699029.739190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146219136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699029.739202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699029.739208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699029/real 1787699029] req@ffff8f518738aa00 x1874531146219136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699029.739214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146219136/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699029.739218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699029.739219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699029.739221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146219136:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699030.634537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699031.658540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699032.682501:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699033.706544:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699034.730528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699034.858556:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858566:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699034.858570:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858580:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858590:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858597:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699034.858632:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146219264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699034.858657:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858661:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699034.858662:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858666:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858669:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858673:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858681:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699034.858693:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146219392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699034.858702:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146219264:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699034.858705:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858710:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699034.858712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858716:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858720:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858723:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858727:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699034.858737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146219520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699034.858746:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699034.858751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858754:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858758:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858761:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858765:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787699034.858767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699034.858778:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146219648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699034.858787:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858790:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699034.858791:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858795:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858799:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858802:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858807:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699034.858808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183647480 x1874531146219264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699034.858821:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146219776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699034.858827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146219264/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699034.858835:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858838:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699034.858840:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858844:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858849:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858854:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787699034.858867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.858872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787699034.858874:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699034.858878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146219264:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787699034.858891:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146219904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699034.858900:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858904:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699034.858905:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858909:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858912:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858915:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858920:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699034.858930:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146220032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699034.858936:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858938:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699034.858939:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858942:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858945:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858949:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858953:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699034.858962:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146220160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699034.858970:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.858973:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699034.858974:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.858977:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6934 00000100:00080000:1.0:1787699034.858980:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699034.858983:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699034.858987:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699034.858996:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146220288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699034.859214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699034.859239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183647100 x1874531146219392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146219392/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699034.859283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219520:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699034.859298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183645180 x1874531146219520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146219520/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699034.859336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699034.859351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183644380 x1874531146219648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146219648/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219648:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699034.859445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219776:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699034.859462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183645c00 x1874531146219776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146219776/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146219776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699034.859499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146219904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699034.859521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183644000 x1874531146219904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146219904/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146219904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699034.859561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146220032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699034.859576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183646680 x1874531146220032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146220032/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146220032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699034.859623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146220160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699034.859638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183647800 x1874531146220160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146220160/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146220160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699034.859678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146220288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699034.859692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699034.859701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff8f5183644700 x1874531146220288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699034.859711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146220288/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699034.859718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699034.859721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699034.859724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146220288:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699035.754593:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699036.778533:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699037.802538:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699038.826577:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699039.850574:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699039.978477:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.978489:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699039.978493:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.978501:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.978510:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978515:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699039.978542:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146220416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699039.978562:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.978564:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699039.978565:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.978568:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.978570:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978572:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.978578:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699039.978585:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146220544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699039.978592:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.978594:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699039.978594:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.978596:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.978598:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978600:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.978603:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699039.978610:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146220672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699039.978615:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.978617:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699039.978618:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.978620:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.978623:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978627:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.978632:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699039.978640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146220416:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699039.978651:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146220800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699039.978665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.978669:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699039.978671:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.978675:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.978679:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978683:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.978689:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699039.978701:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146220928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699039.978710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.978714:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699039.978716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787699039.978723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699039.978729:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.978735:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978740:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.978747:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699039.978763:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146221056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699039.978774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace91880 x1874531146220416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.978812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146220416/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.978858:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699039.978858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699039.978867:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699039.978870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699039.978872:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699039.978877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146220416:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787699039.978883:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00100000:2.0:1787699039.978889:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220544:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699039.978890:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.978902:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.978911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787699039.978916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.978930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace93800 x1874531146220544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699039.978940:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146221184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699039.978942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146220544/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699039.979014:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699039.979015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699039.979024:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699039.979027:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699039.979027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220544:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699039.979041:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00100000:2.0:1787699039.979047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220672:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699039.979048:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.979056:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699039.979061:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699039.979070:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146221312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699039.979077:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.979080:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699039.979081:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.979084:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6939 00000100:00080000:0.0:1787699039.979087:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699039.979093:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699039.979094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787699039.979103:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699039.979112:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146221440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699039.979118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace90380 x1874531146220672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146220672/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699039.979178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220800:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699039.979195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.979202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace92680 x1874531146220800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146220800/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220800:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699039.979221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220928:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699039.979231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.979236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace91500 x1874531146220928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146220928/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146220928:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699039.979259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699039.979269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.979275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace93480 x1874531146221056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146221056/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699039.979292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699039.979302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.979307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace92d80 x1874531146221184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146221184/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699039.979329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699039.979338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.979343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace90e00 x1874531146221312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146221312/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699039.979401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699039.979429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699039.979441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff8f51ace93b80 x1874531146221440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699039.979449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146221440/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699039.979454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699039.979456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699039.979459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146221440:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699040.874542:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699041.898461:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699042.922584:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699043.946548:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699044.970530:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699045.098459:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699045.098467:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699045.098470:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098485:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098494:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098499:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699045.098526:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146221568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699045.098546:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699045.098549:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699045.098551:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098556:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098563:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098567:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098575:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699045.098585:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146221696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699045.098585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146221568:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787699045.098600:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699045.098602:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699045.098604:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098607:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098609:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098612:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098615:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699045.098629:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146221824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699045.098640:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787699045.098640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699045.098643:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699045.098644:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098647:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098650:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098653:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098657:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699045.098665:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146221952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699045.098673:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699045.098676:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699045.098676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace92300 x1874531146221568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699045.098677:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098680:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098684:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098686:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699045.098692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146221568/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699045.098698:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146222080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699045.098701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699045.098705:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699045.098705:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699045.098708:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787699045.098708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146221568:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787699045.098710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098713:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098716:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098718:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098722:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699045.098726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146221696:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699045.098730:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146222208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699045.098738:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699045.098741:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699045.098742:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787699045.098744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699045.098745:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098748:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098751:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699045.098754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace93100 x1874531146221696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699045.098755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699045.098764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146222336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699045.098765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146221696/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.098773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.098776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699045.098779:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699045.098779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146221696:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699045.098782:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699045.098783:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699045.098785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146221824:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699045.098786:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00080000:0.0:1787699045.098789:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098792:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098795:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699045.098801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699045.098803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146222464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699045.098810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace90700 x1874531146221824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699045.098811:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699045.098814:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699045.098816:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699045.098819:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6944 00000100:00100000:2.0:1787699045.098821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146221824/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699045.098822:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699045.098825:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699045.098829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699045.098829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.098832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.098835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146221824:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699045.098837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146222592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699045.098840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146221952:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699045.098856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699045.098865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace90000 x1874531146221952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699045.098875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146221952/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.098883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.098885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.098887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146221952:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699045.098914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146222080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699045.098928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699045.098935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace91180 x1874531146222080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699045.098942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146222080/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.098950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.098953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.098957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146222080:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699045.098972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699045.098990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699045.099001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace91c00 x1874531146222208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699045.099008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146222208/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.099013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.099015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.099017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699045.099020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699045.099032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699045.099037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace92300 x1874531146222336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699045.099043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146222336/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.099047:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.099049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.099051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699045.099070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699045.099081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699045.099087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace93b80 x1874531146222464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699045.099094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146222464/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.099100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.099102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.099104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699045.099108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699045.099116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699045.099121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699045/real 1787699045] req@ffff8f51ace90e00 x1874531146222592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699045.099133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146222592/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699045.099140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699045.099142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699045.099144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146222592:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699045.994549:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699047.018531:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699048.042587:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699049.066510:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699050.090536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699050.218463:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218471:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699050.218474:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218482:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218491:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218497:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699050.218531:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146222720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699050.218557:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218563:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699050.218565:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218568:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218570:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218573:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699050.218581:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699050.218590:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146222848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699050.218599:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218601:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699050.218602:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218605:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218607:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218609:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699050.218615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146222720:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787699050.218618:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699050.218631:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146222976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699050.218638:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218641:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699050.218642:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218644:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218646:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218648:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699050.218651:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699050.218658:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146223104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699050.218664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699050.218668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218669:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218673:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218675:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699050.218678:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699050.218684:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146223232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699050.218691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699050.218694:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218697:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699050.218698:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218702:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218704:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218707:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699050.218710:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699050.218716:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146223360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699050.218722:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218724:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699050.218725:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218728:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00080000:3.0:1787699050.218730:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218732:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699050.218737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f5187388e00 x1874531146222720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699050.218737:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699050.218747:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146223488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699050.218757:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218758:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699050.218760:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699050.218763:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00100000:2.0:1787699050.218764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146222720/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699050.218766:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699050.218776:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699050.218785:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146223616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699050.218786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699050.218795:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699050.218797:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699050.218799:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699050.218813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699050.218819:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6949 00000100:00100000:2.0:1787699050.218822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146222720:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699050.218824:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699050.218830:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699050.218834:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699050.218837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146222848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699050.218845:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146223744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699050.218867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.218884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f5187388000 x1874531146222848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.218896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146222848/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.218905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.218908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.218911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146222848:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699050.218931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146222976:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699050.218950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.218962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f5187389180 x1874531146222976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.218974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146222976/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.218987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.218990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.218993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146222976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699050.218999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146223104:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699050.219017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.219028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f518738a680 x1874531146223104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.219049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146223104/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.219057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.219060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.219063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146223104:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699050.219068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146223232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699050.219087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.219106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f5187389f80 x1874531146223232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.219118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146223232/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.219133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.219136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.219139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146223232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699050.219144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699050.219162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.219172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f518738a300 x1874531146223360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.219181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146223360/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.219190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.219194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.219197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699050.219202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223488:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699050.219217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.219228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f518738b100 x1874531146223488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.219238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146223488/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.219245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.219248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.219251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699050.219256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699050.219273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.219285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f518738b480 x1874531146223616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.219296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146223616/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.219303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.219306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.219309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699050.219314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699050.219331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699050.219342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699050/real 1787699050] req@ffff8f518738aa00 x1874531146223744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699050.219376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146223744/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699050.219385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699050.219389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699050.219392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146223744:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699051.114528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699052.138493:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699053.162537:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699054.186486:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699055.210540:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699055.338435:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338445:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699055.338448:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338458:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338468:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338474:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699055.338508:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146223872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699055.338539:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338543:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699055.338545:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338549:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338553:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338557:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338565:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699055.338575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146223872:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699055.338578:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146224000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699055.338592:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338593:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699055.338594:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338596:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338601:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699055.338613:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146224128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699055.338619:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699055.338622:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338624:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338626:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787699055.338631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699055.338631:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338637:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699055.338644:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146224256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699055.338650:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338652:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699055.338653:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338655:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338658:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338660:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338663:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699055.338670:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146224384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699055.338675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f518738ad80 x1874531146223872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699055.338677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338682:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699055.338685:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338691:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338695:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338698:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338701:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699055.338709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146223872/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699055.338711:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146224512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699055.338752:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699055.338753:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338760:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699055.338761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699055.338763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699055.338769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146223872:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699055.338771:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338779:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338783:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338788:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699055.338801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224000:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699055.338802:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146224640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699055.338819:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338821:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699055.338822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338826:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338829:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699055.338831:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699055.338836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699055.338836:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699055.338850:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146224768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699055.338856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f5187389500 x1874531146224000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699055.338866:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699055.338871:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699055.338874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146224000/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699055.338874:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699055.338883:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6954 00000100:00080000:3.0:1787699055.338889:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699055.338891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.338895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.338898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224000:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699055.338904:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699055.338909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224128:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699055.338912:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699055.338927:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146224896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699055.338935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.338952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f5187389c00 x1874531146224128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.338965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146224128/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.338973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.338976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.338979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699055.338985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224256:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699055.339001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.339008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f5187388380 x1874531146224256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.339015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146224256/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.339020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.339022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.339024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224256:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699055.339027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699055.339036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.339041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f5187388a80 x1874531146224384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.339047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146224384/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.339051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.339052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.339054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146224384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699055.339057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699055.339065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.339070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f518738b800 x1874531146224512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.339077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146224512/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.339082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.339083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.339085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699055.339098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699055.339109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.339115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f518738bb80 x1874531146224640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.339121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146224640/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.339126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.339128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.339129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699055.339132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699055.339141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.339149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f518738ad80 x1874531146224768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.339159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146224768/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.339172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.339175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.339178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699055.339183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699055.339201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699055.339212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699055/real 1787699055] req@ffff8f518738aa00 x1874531146224896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699055.339222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146224896/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699055.339229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699055.339232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699055.339236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146224896:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699056.234603:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699057.258571:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699058.282641:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699059.306502:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699060.330553:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699060.458470:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458480:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699060.458484:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.458495:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.458507:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458513:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699060.458561:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146225024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699060.458595:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458598:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699060.458600:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.458605:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.458613:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458618:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699060.458641:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699060.458660:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146225152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699060.458673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146225024:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699060.458677:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458690:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699060.458692:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.458697:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.458701:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458705:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699060.458711:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699060.458726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146225280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699060.458736:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458739:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699060.458741:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.458745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.458758:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458763:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699060.458769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699060.458776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699060.458785:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146225408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699060.458796:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699060.458802:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.458806:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.458811:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458816:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699060.458822:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699060.458830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183644e00 x1874531146225024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699060.458839:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146225536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699060.458855:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699060.458865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.458871:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00100000:2.0:1787699060.458873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146225024/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699060.458876:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458890:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699060.458894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787699060.458897:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787699060.458901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.458909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146225024:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787699060.458926:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146225664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699060.458937:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.458940:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699060.458942:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699060.458945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225152:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699060.458951:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.458956:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.458961:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699060.458967:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787699060.458985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699060.458994:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146225792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699060.459004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183646d80 x1874531146225152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699060.459012:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.459018:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699060.459021:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.459041:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.459049:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699060.459053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146225152/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699060.459062:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699060.459069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787699060.459073:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699060.459074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225152:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699060.459096:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146225920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699060.459108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699060.459111:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699060.459122:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699060.459124:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699060.459130:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6960 00000100:00080000:1.0:1787699060.459134:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699060.459137:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699060.459143:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699060.459156:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146226048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699060.459161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183646a00 x1874531146225280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146225280/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699060.459219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225408:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699060.459234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183647b80 x1874531146225408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146225408/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225408:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699060.459263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699060.459281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183644700 x1874531146225536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146225536/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146225536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699060.459320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146225664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699060.459346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183647800 x1874531146225664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146225664/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146225664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699060.459440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146225792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699060.459460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183646680 x1874531146225792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146225792/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146225792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699060.459494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146225920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699060.459511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183644000 x1874531146225920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146225920/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146225920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699060.459551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146226048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699060.459567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699060.459576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699060/real 1787699060] req@ffff8f5183645c00 x1874531146226048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699060.459587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146226048/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699060.459595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699060.459596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699060.459607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146226048:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699061.354578:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699062.378540:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699063.402532:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699064.426532:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699065.450558:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699065.578467:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578477:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699065.578481:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699065.578490:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578499:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578506:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699065.578551:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146226176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699065.578580:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578584:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699065.578586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699065.578590:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578593:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578596:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699065.578604:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699065.578622:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146226304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699065.578631:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578633:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699065.578635:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699065.578638:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578641:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578643:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699065.578648:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699065.578658:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146226432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699065.578667:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578670:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699065.578674:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699065.578674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146226176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699065.578682:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578685:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578688:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699065.578693:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699065.578703:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146226560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699065.578710:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578713:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699065.578715:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699065.578718:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578728:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578733:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699065.578736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787699065.578739:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699065.578754:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146226688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699065.578762:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578765:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699065.578772:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699065.578775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183644380 x1874531146226176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699065.578779:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578786:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578792:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699065.578797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146226176/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699065.578798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787699065.578813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787699065.578814:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146226816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699065.578824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787699065.578826:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578831:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699065.578833:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699065.578834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146226176:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699065.578838:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578845:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699065.578850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226304:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699065.578853:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699065.578859:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699065.578868:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146226944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699065.578875:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578881:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787699065.578882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699065.578885:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699065.578892:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00100000:2.0:1787699065.578896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183645180 x1874531146226304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699065.578897:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578905:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699065.578910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146226304/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699065.578912:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699065.578923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.578927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699065.578931:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146227072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699065.578935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226304:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699065.578950:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699065.578953:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699065.578955:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699065.578959:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6965 00000100:00080000:1.0:1787699065.578963:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699065.578969:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699065.578969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226432:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787699065.578975:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699065.578986:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146227200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699065.579000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183647100 x1874531146226432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146226432/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226432:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699065.579042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226560:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699065.579059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183647480 x1874531146226560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146226560/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226560:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699065.579098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226688:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699065.579470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183645c00 x1874531146226688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146226688/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146226688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699065.579514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146226816:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699065.579534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183644000 x1874531146226816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146226816/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146226816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699065.579589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146226944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699065.579615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183646680 x1874531146226944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146226944/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146226944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699065.579654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146227072:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699065.579672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183647800 x1874531146227072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146227072/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579702:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146227072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699065.579723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146227200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699065.579742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699065.579753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699065/real 1787699065] req@ffff8f5183644700 x1874531146227200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699065.579764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146227200/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699065.579772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699065.579775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699065.579778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146227200:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699066.474632:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699067.498623:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699068.522544:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699069.546534:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699070.570564:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699070.698494:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698501:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699070.698504:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698511:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698526:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698533:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699070.698564:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146227328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698588:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699070.698589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698592:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698594:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698597:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699070.698618:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146227456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698625:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698627:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699070.698628:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698631:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698633:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698635:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698638:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699070.698650:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146227584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699070.698653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146227328:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699070.698659:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698665:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699070.698667:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698670:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698672:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698674:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698676:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699070.698685:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146227712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698690:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698692:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699070.698693:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698696:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698698:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698700:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698704:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699070.698718:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146227840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698727:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698730:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699070.698733:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698737:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698742:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698750:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699070.698751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698761:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699070.698773:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146227968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698782:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698784:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699070.698787:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698789:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698791:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698795:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698798:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699070.698804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146228096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698810:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698812:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699070.698814:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699070.698822:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00100000:2.0:1787699070.698824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183647b80 x1874531146227328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699070.698828:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698834:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698842:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699070.698855:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146228224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699070.698861:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699070.698863:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699070.698865:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699070.698868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146227328/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699070.698869:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6970 00000100:00080000:1.0:1787699070.698875:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699070.698881:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699070.698886:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699070.698896:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146228352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699070.698909:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.698918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.698924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146227328:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699070.698934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227456:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699070.698961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.698974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183646a00 x1874531146227456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.698986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146227456/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227456:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699070.699043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227584:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699070.699061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183646d80 x1874531146227584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146227584/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699070.699103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227712:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699070.699119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183644e00 x1874531146227712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146227712/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227712:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699070.699162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699070.699178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183644700 x1874531146227840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146227840/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146227840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699070.699218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146227968:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699070.699235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183647800 x1874531146227968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146227968/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146227968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699070.699278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146228096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699070.699294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183646680 x1874531146228096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146228096/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146228096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699070.699336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146228224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699070.699352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183644000 x1874531146228224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146228224/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146228224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699070.699427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146228352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699070.699444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699070.699467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699070/real 1787699070] req@ffff8f5183645c00 x1874531146228352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699070.699480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146228352/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699070.699489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699070.699492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699070.699496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146228352:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699071.594593:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699072.618553:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699073.642485:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699074.666584:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699075.690614:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699075.818497:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818506:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699075.818510:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818519:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818533:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818541:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699075.818586:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146228480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699075.818617:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699075.818623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818628:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818632:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818636:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.818646:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699075.818659:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146228608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699075.818670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146228480:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699075.818672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818679:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699075.818681:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818685:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818689:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818692:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.818698:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699075.818712:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146228736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699075.818722:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818726:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699075.818728:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818732:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818736:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818740:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.818746:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699075.818753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699075.818759:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146228864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699075.818770:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818774:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699075.818776:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818781:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818785:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818788:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.818793:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699075.818802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f518738b480 x1874531146228480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699075.818807:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146228992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699075.818824:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818829:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787699075.818831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146228480/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699075.818832:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818839:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818845:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818850:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699075.818851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787699075.818858:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787699075.818859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.818866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146228480:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699075.818873:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146229120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699075.818883:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818886:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699075.818888:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818893:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.818895:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699075.818897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699075.818903:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.818909:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699075.818921:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146229248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699075.818930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699075.818934:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.818938:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699075.818941:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699075.818946:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00100000:2.0:1787699075.818947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f518738b100 x1874531146228608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699075.818951:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.818958:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.818965:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699075.818968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146228608/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699075.818979:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146229376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699075.818981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.818989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.818998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228608:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699075.818998:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699075.819005:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699075.819009:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699075.819011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699075.819016:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6975 00000100:00080000:3.0:1787699075.819021:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699075.819024:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699075.819029:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699075.819045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699075.819055:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146229504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699075.819060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f518738a300 x1874531146228736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146228736/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699075.819114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228864:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699075.819135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699075.819144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f5187389f80 x1874531146228864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146228864/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228864:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699075.819175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228992:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699075.819197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699075.819206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f518738a680 x1874531146228992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146228992/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146228992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699075.819236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229120:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699075.819252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699075.819261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f5187389180 x1874531146229120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146229120/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699075.819304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229248:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699075.819321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699075.819331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f5187388000 x1874531146229248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146229248/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699075.819410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699075.819430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699075.819440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f5187388e00 x1874531146229376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146229376/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699075.819472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229504:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699075.819489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699075.819500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699075/real 1787699075] req@ffff8f518738b480 x1874531146229504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699075.819512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146229504/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699075.819520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699075.819523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699075.819527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146229504:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699076.714606:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699077.738553:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699078.762522:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699079.786514:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699080.810506:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699080.938550:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938560:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699080.938563:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938572:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938582:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938587:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699080.938621:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146229632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.938653:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938657:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699080.938658:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938663:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938667:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938671:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938679:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699080.938692:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146229760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.938715:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938718:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699080.938719:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938723:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00100000:2.0:1787699080.938726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146229632:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699080.938726:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938729:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938733:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699080.938744:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146229888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.938759:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938762:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699080.938763:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938766:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938768:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938771:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938775:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699080.938784:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146230016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.938791:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938794:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699080.938796:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938799:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938802:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938805:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938808:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699080.938816:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146230144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699080.938818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146229760:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699080.938825:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938828:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699080.938830:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938834:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938837:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938840:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938844:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699080.938849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146229888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699080.938854:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146230272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.938862:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938866:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787699080.938867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146230016:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699080.938867:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938871:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938873:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938877:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938881:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699080.938887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146230144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699080.938890:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146230400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699080.938904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230272:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699080.938904:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938907:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699080.938909:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699080.938912:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938916:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938919:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699080.938923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230400:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699080.938923:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699080.938933:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146230528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.938943:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699080.938946:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699080.938947:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699080.938950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230528:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699080.938950:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6980 00000100:00080000:3.0:1787699080.938953:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699080.938956:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699080.938961:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699080.938970:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146230656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699080.939011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230656:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699081.834551:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699082.858509:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699083.882492:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787699084.010615:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:3.0:1787699084.010651:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787699084.010666:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010717:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010730:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010738:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010746:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010753:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010760:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010769:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:3.0:1787699084.010776:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787699084.010938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f518738b480 x1874531146230656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.010967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146230656/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.010982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.010988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.010993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699084.011001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f5187389500 x1874531146230528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146230528/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699084.011032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f5187389c00 x1874531146230400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146230400/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699084.011059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f5187388380 x1874531146230272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146230272/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146230272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699084.011102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f518738bb80 x1874531146229888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146229888/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146229888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699084.011134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f518738b800 x1874531146230016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146230016/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146230016:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699084.011165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f5187388a80 x1874531146230144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146230144/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146230144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699084.011195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f518738aa00 x1874531146229632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146229632/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146229632:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699084.011258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699084] req@ffff8f518738ad80 x1874531146229760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699084.011272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146229760/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699084.011280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699084.011288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699084.011294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146229760:192.168.201.125@tcp:38:kworker.0 00010000:00010000:0.0:1787699084.906506:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699085.930522:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699086.058485:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058494:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699086.058498:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058508:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058519:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058525:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699086.058555:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146230784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058586:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058590:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699086.058591:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058595:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058599:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058602:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058609:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699086.058620:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146230912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058629:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058631:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699086.058633:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058636:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058638:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058641:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058645:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699086.058652:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146231040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058665:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058669:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699086.058670:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058673:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058676:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058679:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058683:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699086.058691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146231168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058698:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058700:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699086.058702:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058705:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058708:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058711:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058714:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699086.058723:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146231296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058731:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699086.058735:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058739:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058741:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058745:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058749:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699086.058758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146231424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058766:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058769:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699086.058770:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058774:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058776:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058779:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058783:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699086.058791:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146231552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058799:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058803:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699086.058804:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058807:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058810:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058813:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058816:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699086.058823:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146231680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699086.058831:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699086.058834:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699086.058836:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699086.058839:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6985 00000100:00080000:1.0:1787699086.058842:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699086.058845:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699086.058850:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699086.058859:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146231808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699086.058954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146230784:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699086.059025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183647480 x1874531146230784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146230784/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146230784:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699086.059113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146230912:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699086.059136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183647100 x1874531146230912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146230912/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146230912:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699086.059176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146231040:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699086.059209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183645180 x1874531146231040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146231040/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146231040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699086.059248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146231168:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699086.059266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183644380 x1874531146231168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146231168/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146231168:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699086.059304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146231296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699086.059322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183645c00 x1874531146231296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146231296/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146231296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699086.059402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699086.059421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183644000 x1874531146231424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146231424/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699086.059465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699086.059484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183646680 x1874531146231552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146231552/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699086.059529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699086.059548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183647800 x1874531146231680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146231680/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699086.059590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699086.059608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699086.059619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699086/real 1787699086] req@ffff8f5183644700 x1874531146231808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699086.059629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146231808/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699086.059637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699086.059639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699086.059642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146231808:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699086.954516:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699087.978534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699089.002527:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699090.026533:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699091.050489:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699091.178464:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178474:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699091.178478:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178488:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178503:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178508:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699091.178539:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146231936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699091.178563:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178567:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699091.178569:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178573:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178577:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178581:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178589:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699091.178605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146231936:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699091.178606:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146232064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699091.178620:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178622:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699091.178623:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178626:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178628:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178630:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178633:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699091.178641:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146232192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699091.178647:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178649:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699091.178650:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178652:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178654:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178656:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178659:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699091.178668:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146232320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699091.178670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699091.178681:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178683:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699091.178685:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178687:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178689:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178690:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178692:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699091.178698:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146232448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699091.178703:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178705:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699091.178706:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178708:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178710:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178712:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178715:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699091.178722:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146232576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699091.178728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace92d80 x1874531146231936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699091.178730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178734:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699091.178736:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178741:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178745:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699091.178749:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178753:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699091.178759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146232704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699091.178764:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178766:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699091.178767:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178769:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178771:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699091.178773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146231936/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699091.178775:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178779:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699091.178788:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146232832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699091.178792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699091.178795:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699091.178798:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699091.178799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699091.178800:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699091.178805:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6990 00000100:00080000:0.0:1787699091.178808:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699091.178809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146231936:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787699091.178812:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699091.178817:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699091.178825:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146232960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699091.178840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232064:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699091.178865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.178879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace93480 x1874531146232064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.178889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146232064/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.178894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.178896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.178898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232064:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699091.178901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699091.178914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.178920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace91500 x1874531146232192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.178927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146232192/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.178931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.178933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.178935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699091.178938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232320:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699091.178946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.178951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace92680 x1874531146232320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.178957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146232320/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.178962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.178964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.178965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232320:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699091.178968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699091.178976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.178980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace90380 x1874531146232448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.178986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146232448/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.178990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.178992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.178993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146232448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699091.178996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699091.179009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.179018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace93800 x1874531146232576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.179028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146232576/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.179035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.179038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.179042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699091.179048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699091.179065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.179074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace91880 x1874531146232704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.179085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146232704/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.179093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.179095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.179098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699091.179103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699091.179133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.179144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace90e00 x1874531146232832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.179154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146232832/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.179161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.179164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.179167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699091.179172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699091.179186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699091.179196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699091/real 1787699091] req@ffff8f51ace93b80 x1874531146232960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699091.179206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146232960/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699091.179214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699091.179217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699091.179220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146232960:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699092.074494:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699093.098535:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699094.122531:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699095.146495:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699096.170609:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699096.298446:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298456:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699096.298460:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699096.298470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:3.0:1787699096.298480:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298487:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699096.298522:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146233088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699096.298551:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298559:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699096.298561:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699096.298566:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:3.0:1787699096.298570:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298574:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699096.298582:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699096.298590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146233088:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699096.298597:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146233216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699096.298609:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298613:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699096.298615:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699096.298619:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:3.0:1787699096.298633:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298637:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699096.298642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699096.298644:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699096.298661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146233344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699096.298672:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298675:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699096.298677:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699096.298681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f5187388e00 x1874531146233088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699096.298684:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:3.0:1787699096.298690:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298697:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699096.298703:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699096.298705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146233088/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699096.298717:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146233472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699096.298718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699096.298727:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298733:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699096.298735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699096.298737:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699096.298742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146233088:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699096.298745:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:3.0:1787699096.298751:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298756:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699096.298761:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699096.298771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699096.298774:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146233600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699096.298786:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298790:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699096.298792:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699096.298796:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6995 00000800:00000100:2.0:1787699096.298800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699096.298802:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298809:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699096.298813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f5187388000 x1874531146233216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699096.298815:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699096.298828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146233216/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699096.298832:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146233728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699096.298843:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298849:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699096.298852:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699096.298859:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:2.0:1787699096.298865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699096.298865:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.298871:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699096.298873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787699096.298878:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699096.298880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233216:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699096.298890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233344:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699096.298897:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146233856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699096.298907:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699096.298910:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699096.298914:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699096.298918:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00080000:3.0:1787699096.298921:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787699096.298922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699096.298926:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699096.298931:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699096.298943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f5187389180 x1874531146233344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699096.298944:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146233984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699096.298965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146233344/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699096.298973:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699096.298978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699096.298979:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699096.298983:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699096.298986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699096.298991:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 6995 00000100:00100000:2.0:1787699096.298995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233344:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699096.298997:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699096.299004:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233472:192.168.201.125@tcp:8:kworker.0 00000008:00100000:3.0:1787699096.299011:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699096.299036:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146234112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699096.299044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299058:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f518738a680 x1874531146233472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699096.299066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146233472/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699096.299072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699096.299074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699096.299076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233472:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699096.299087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699096.299097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f5187389f80 x1874531146233600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699096.299109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146233600/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699096.299114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699096.299115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699096.299117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146233600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699096.299120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146233728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699096.299129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f518738a300 x1874531146233728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699096.299140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146233728/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699096.299145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699096.299148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699096.299151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146233728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699096.299155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146233856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699096.299170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f518738b100 x1874531146233856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699096.299192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146233856/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699096.299198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699096.299200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699096.299202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146233856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699096.299206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146233984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699096.299218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f5187388e00 x1874531146233984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699096.299231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146233984/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699096.299235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699096.299237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699096.299239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146233984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699096.299241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146234112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699096.299249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699096.299254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699096/real 1787699096] req@ffff8f518738ad80 x1874531146234112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699096.299260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146234112/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699096.299269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699096.299271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699096.299273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146234112:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699097.194536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699098.218517:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699099.242529:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699100.266493:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699101.290558:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699101.418725:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.418734:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699101.418737:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.418746:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.418757:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.418764:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699101.418799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146234240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699101.418820:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.418823:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699101.418825:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.418830:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.418834:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.418838:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.418847:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699101.418859:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146234368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699101.418868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146234240:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699101.418869:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.418872:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699101.418873:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.418877:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.418879:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.418882:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.418886:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699101.418895:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146234496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699101.418902:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.418905:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699101.418906:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.418909:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.418912:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.418914:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.418918:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699101.418925:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146234624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699101.418949:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787699101.418950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699101.418952:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699101.418953:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.418957:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.418959:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.418962:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.418966:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699101.418976:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146234752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699101.418984:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.418987:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699101.418988:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.418991:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.418994:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699101.418996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183644e00 x1874531146234240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699101.418998:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.419001:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699101.419011:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146234880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699101.419011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146234240/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699101.419020:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699101.419021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699101.419023:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699101.419025:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699101.419025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:1.0:1787699101.419028:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00100000:2.0:1787699101.419028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146234240:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699101.419031:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.419034:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.419037:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699101.419049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234368:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699101.419053:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146235008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699101.419065:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.419067:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699101.419069:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787699101.419069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699101.419072:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.419076:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699101.419079:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699101.419083:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699101.419093:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146235136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699101.419093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183646d80 x1874531146234368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699101.419102:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699101.419105:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699101.419105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146234368/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699101.419107:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699101.419110:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7001 00000100:00080000:1.0:1787699101.419113:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699101.419113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699101.419116:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699101.419116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234368:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787699101.419120:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699101.419125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699101.419128:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146235264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699101.419141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183646a00 x1874531146234496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146234496/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699101.419181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234624:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699101.419199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183647b80 x1874531146234624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146234624/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234624:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699101.419235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234752:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699101.419251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183644700 x1874531146234752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146234752/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146234752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699101.419298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146234880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699101.419317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183647800 x1874531146234880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146234880/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146234880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699101.419411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146235008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699101.419431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183646680 x1874531146235008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146235008/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146235008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699101.419479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146235136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699101.419496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183644000 x1874531146235136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146235136/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146235136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699101.419534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146235264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699101.419550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699101.419560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699101/real 1787699101] req@ffff8f5183645c00 x1874531146235264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699101.419570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146235264/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699101.419576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699101.419579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699101.419582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146235264:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699102.314506:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699103.338498:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699104.362551:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699105.386573:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699106.410544:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699106.538447:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538457:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699106.538461:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699106.538470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538480:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699106.538489:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699106.538522:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146235392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699106.538548:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538552:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699106.538557:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699106.538563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538567:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699106.538573:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699106.538573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146235392:192.168.201.125@tcp:250:kworker.0 00000008:00100000:1.0:1787699106.538586:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699106.538606:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146235520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699106.538618:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538621:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699106.538623:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787699106.538628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699106.538631:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538638:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699106.538642:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699106.538648:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699106.538661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146235648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699106.538671:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538674:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787699106.538677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183644380 x1874531146235392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.538678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699106.538687:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538691:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699106.538697:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699106.538703:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699106.538704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146235392/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699106.538714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787699106.538718:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146235776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699106.538719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.538726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146235392:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699106.538739:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538743:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699106.538745:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699106.538749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235520:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699106.538753:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538757:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699106.538761:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699106.538766:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699106.538770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699106.538785:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146235904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699106.538785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183645180 x1874531146235520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.538813:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538817:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699106.538819:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699106.538819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146235520/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.538827:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538832:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699106.538838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699106.538839:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699106.538857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787699106.538861:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699106.538862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235520:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699106.538873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699106.538875:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146236032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699106.538884:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538887:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699106.538888:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699106.538891:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538894:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699106.538897:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699106.538900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787699106.538904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699106.538910:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146236160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699106.538917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183647100 x1874531146235648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.538918:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538925:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699106.538927:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699106.538933:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.538938:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699106.538940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146235648/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.538944:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699106.538952:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699106.538954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.538959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.538962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699106.538969:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146236288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699106.538969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235776:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787699106.538984:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699106.538989:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787699106.538990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699106.538995:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699106.538999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183647480 x1874531146235776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.539001:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7006 00000100:00080000:1.0:1787699106.539009:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699106.539013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146235776/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699106.539016:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699106.539025:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699106.539026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.539032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.539034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235776:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699106.539040:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146236416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699106.539048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235904:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699106.539062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699106.539072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183644380 x1874531146235904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699106.539079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146235904/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699106.539084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.539086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.539087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146235904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699106.539090:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236032:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699106.539098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699106.539103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183645c00 x1874531146236032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699106.539109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146236032/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699106.539114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.539116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.539117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236032:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699106.539120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699106.539128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699106.539133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183644000 x1874531146236160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699106.539139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146236160/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699106.539143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.539145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.539147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699106.539150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236288:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699106.539164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699106.539173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183646680 x1874531146236288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699106.539183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146236288/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699106.539191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.539193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.539196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236288:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699106.539209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699106.539221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699106.539228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699106/real 1787699106] req@ffff8f5183647800 x1874531146236416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699106.539234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146236416/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699106.539239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699106.539240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699106.539242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146236416:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699107.434542:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699108.458547:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699109.482540:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699110.506530:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699111.530561:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699111.658478:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699111.658488:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699111.658492:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.658517:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.658528:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658535:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699111.658571:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146236544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699111.658596:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699111.658599:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699111.658601:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.658606:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.658610:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658614:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.658621:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699111.658633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146236544:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699111.658637:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146236672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699111.658646:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699111.658647:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699111.658649:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.658652:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.658656:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658660:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.658670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699111.658682:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146236800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699111.658687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699111.658702:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699111.658705:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699111.658707:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.658712:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.658716:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658720:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.658726:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699111.658735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace92300 x1874531146236544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699111.658739:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146236928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699111.658783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146236544/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699111.658789:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699111.658793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.658797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699111.658800:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699111.658800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146236544:192.168.201.125@tcp:250:kworker.0 00000100:00080000:0.0:1787699111.658804:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.658813:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.658817:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658821:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.658826:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699111.658826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146236672:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699111.658844:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146237056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699111.658855:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699111.658859:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699111.658861:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.658865:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7011 00000800:00000100:2.0:1787699111.658869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699111.658871:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658878:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.658883:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699111.658884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace91c00 x1874531146236672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699111.658897:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146237184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699111.658901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146236672/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.658911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.658918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:0.0:1787699111.658921:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699111.658924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146236672:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699111.658928:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699111.658931:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699111.658933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146236800:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699111.658938:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.658951:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.658956:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699111.658960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:0.0:1787699111.658964:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699111.658971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace91180 x1874531146236800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699111.658979:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146237312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699111.658981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146236800/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.658993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.658998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146236800:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699111.659008:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699111.659012:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146236928:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787699111.659014:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699111.659017:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699111.659023:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:0.0:1787699111.659028:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699111.659032:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.659037:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699111.659037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699111.659051:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146237440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699111.659055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace90000 x1874531146236928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699111.659072:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699111.659073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146236928/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699111.659079:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699111.659082:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699111.659084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699111.659088:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7011 00000100:00080000:2.0:1787699111.659090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659094:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146236928:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787699111.659109:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699111.659109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146237056:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787699111.659117:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699111.659123:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699111.659135:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146237568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699111.659138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699111.659151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace90700 x1874531146237056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699111.659162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146237056/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.659167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.659169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146237056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699111.659179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237184:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699111.659197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699111.659208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace93100 x1874531146237184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699111.659219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146237184/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.659227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.659231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237184:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699111.659237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699111.659250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699111.659257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace92300 x1874531146237312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699111.659264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146237312/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.659268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.659270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699111.659275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237440:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699111.659283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699111.659288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace93b80 x1874531146237440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699111.659295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146237440/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.659299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.659300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237440:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699111.659311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699111.659320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699111.659326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699111/real 1787699111] req@ffff8f51ace90e00 x1874531146237568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699111.659332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146237568/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699111.659336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699111.659338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699111.659339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146237568:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699112.554528:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699113.578526:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699114.602528:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699115.626539:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699116.650659:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699116.780608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.780620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699116.780625:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.780637:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.780650:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.780658:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699116.780741:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146237696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699116.780813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146237696:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699116.780892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699116.780927:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.780931:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699116.780933:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.780938:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.780942:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.780947:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.780958:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699116.780972:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146237824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699116.781050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f518738aa00 x1874531146237696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.781070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146237696/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.781082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.781086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.781090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146237696:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699116.781128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146237824:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699116.781153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699116.781167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f5187388a80 x1874531146237824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.781404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146237824/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.781414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.781418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.781421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146237824:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699116.782497:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.782502:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699116.782504:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.782509:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.782515:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.782519:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.782527:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699116.782548:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146237952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699116.782594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146237952:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699116.782628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699116.782652:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.782656:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699116.782658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.782663:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.782668:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.782683:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.782690:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699116.782702:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146238080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699116.782744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f518738b800 x1874531146237952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.782757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146237952/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.782766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.782769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.782773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146237952:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699116.782789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146238080:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699116.782812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699116.782825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f518738bb80 x1874531146238080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.782836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146238080/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.782843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.782847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.782850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146238080:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699116.782931:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.782935:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699116.782937:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.782942:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.782947:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.782951:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.782957:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699116.782971:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146238208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699116.783017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146238208:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699116.783041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699116.783061:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.783065:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699116.783067:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.783072:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.783076:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.783080:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.783086:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699116.783244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f5187388380 x1874531146238208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.783257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146238208/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.783266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.783269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.783272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146238208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699116.783317:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146238336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699116.783341:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.783345:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699116.783348:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.783351:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.783373:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.783377:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.783382:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699116.783394:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146238464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699116.783403:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.783406:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699116.783408:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.783412:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.783415:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.783420:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.783424:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699116.783434:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146238592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699116.783443:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699116.783446:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699116.783448:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699116.783452:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7016 00000100:00080000:3.0:1787699116.783456:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699116.783459:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699116.783464:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699116.783474:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146238720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699116.783539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238336:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699116.783563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699116.783578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f5187389c00 x1874531146238336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.783590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146238336/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.783599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.783602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.783605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238336:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699116.783610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699116.783628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699116.783639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f5187389500 x1874531146238464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.783650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146238464/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.783658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.783661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.783664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699116.783670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238592:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699116.783702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699116.783712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f518738b480 x1874531146238592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.783722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146238592/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.783736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.783739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.783742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238592:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699116.783749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699116.783768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699116.783779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699116/real 1787699116] req@ffff8f5187388380 x1874531146238720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699116.783790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146238720/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699116.783798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699116.783801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699116.783804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146238720:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699117.676142:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699118.698584:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699119.722536:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699120.746534:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699121.770502:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699121.898464:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.898473:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699121.898476:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.898485:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.898496:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.898503:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699121.898537:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146238848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699121.898562:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.898566:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699121.898567:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.898571:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.898575:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.898578:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.898602:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699121.898629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146238848:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699121.898674:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146238976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699121.898691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699121.898697:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.898704:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699121.898707:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.898712:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.898716:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.898720:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.898725:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699121.898737:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146239104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699121.898745:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.898749:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699121.898751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.898756:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00100000:2.0:1787699121.898757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f518738bb80 x1874531146238848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699121.898760:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.898768:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.898782:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699121.898783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146238848/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.898797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699121.898799:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146239232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699121.898805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699121.898817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699121.898819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146238848:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699121.898820:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699121.898825:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.898831:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.898835:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.898839:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.898845:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699121.898849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146238976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699121.898860:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146239360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699121.898873:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.898876:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699121.898879:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787699121.898880:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699121.898888:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.898892:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.898896:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.898900:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699121.898913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f518738b800 x1874531146238976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699121.898914:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146239488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699121.898946:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699121.898952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146238976/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699121.898952:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699121.898954:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.898963:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:2.0:1787699121.898969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.898974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.898978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146238976:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699121.898984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146239104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699121.899004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699121.899030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f5187388a80 x1874531146239104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699121.899041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146239104/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.899049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146239104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699121.899061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146239232:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699121.899081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699121.899092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f518738aa00 x1874531146239232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699121.899115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146239232/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.899122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146239232:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699121.899139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146239360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699121.899160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699121.899165:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.899173:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699121.899174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f518738ad80 x1874531146239360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699121.899181:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699121.899191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146239360/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699121.899199:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146239616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699121.899203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146239360:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699121.899218:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.899226:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787699121.899229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239488:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699121.899230:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.899239:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.899242:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.899254:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.899259:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699121.899271:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146239744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699121.899274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699121.899284:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699121.899287:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699121.899289:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699121.899294:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7021 00000100:00080000:3.0:1787699121.899297:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699121.899301:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699121.899306:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699121.899318:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146239872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699121.899319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f5187388e00 x1874531146239488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699121.899351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146239488/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.899392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239488:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699121.899413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699121.899435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699121.899447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f518738b100 x1874531146239616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699121.899457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146239616/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.899464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699121.899476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239744:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699121.899503:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699121.899515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f518738a300 x1874531146239744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699121.899527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146239744/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.899534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239744:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699121.899555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699121.899573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699121.899597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699121] req@ffff8f5187389f80 x1874531146239872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699121.899608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146239872/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699121.899615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699121.899617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699121.899620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146239872:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699122.794619:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699123.818660:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699124.842526:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699125.866695:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699126.890634:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699127.018961:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.018972:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699127.018976:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.018983:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.018992:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.018997:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699127.019044:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146240000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699127.019078:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019081:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699127.019083:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019087:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019091:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.019094:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019104:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699127.019118:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146240128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699127.019129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146240000:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699127.019129:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019136:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699127.019140:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019144:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019147:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.019151:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019156:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699127.019168:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146240256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699127.019176:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019178:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699127.019180:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019182:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019186:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.019189:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699127.019198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019202:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699127.019218:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146240384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699127.019227:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019230:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699127.019232:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019236:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019240:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.019243:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019247:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699127.019255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f518738a680 x1874531146240000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699127.019258:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146240512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699127.019279:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019283:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699127.019284:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019288:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019290:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699127.019294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146240000/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699127.019295:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019303:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787699127.019313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.019318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699127.019321:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146240640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699127.019326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146240000:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699127.019539:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019542:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699127.019544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019549:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019617:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.019625:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699127.019644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240128:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699127.019646:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146240768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699127.019661:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699127.019663:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699127.019664:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019668:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:3.0:1787699127.019678:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787699127.019682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699127.019685:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699127.019691:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699127.019702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f5187389180 x1874531146240128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699127.019704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146240896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699127.019731:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699127.019736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146240128/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699127.019736:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699127.019738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699127.019747:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7026 00000100:00080000:2.0:1787699127.019756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.019760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.019765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240128:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699127.019765:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699127.019774:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699127.019777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240256:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699127.019781:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699127.019796:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146241024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699127.019806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.019822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f5187388000 x1874531146240256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.019834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146240256/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.019841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.019864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.019869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699127.019875:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240384:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699127.019909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.019921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f5187389f80 x1874531146240384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.019932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146240384/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.019939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.019942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.019945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240384:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699127.019963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699127.019981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.019991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f518738a300 x1874531146240512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.020001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146240512/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.020008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.020011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.020014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146240512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699127.020030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146240640:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699127.020048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.020059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f518738b100 x1874531146240640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.020070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146240640/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.020078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.020081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.020084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146240640:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699127.020089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146240768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699127.020106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.020116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f5187388e00 x1874531146240768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.020127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146240768/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.020134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.020138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.020140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146240768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699127.020146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146240896:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699127.020163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.020173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f518738ad80 x1874531146240896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.020183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146240896/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.020191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.020194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.020197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146240896:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699127.020202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146241024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699127.020219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699127.020229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699127/real 1787699127] req@ffff8f518738aa00 x1874531146241024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699127.020239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146241024/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699127.020247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699127.020249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699127.020252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146241024:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699127.914714:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699128.938512:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699129.963115:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699130.986624:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699132.010528:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699132.138504:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138517:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699132.138522:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138532:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138544:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138552:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699132.138591:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146241152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699132.138624:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138628:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699132.138630:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138635:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138639:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138643:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138654:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699132.138667:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146241280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699132.138678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699132.138680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146241152:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699132.138690:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699132.138693:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138697:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138700:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138704:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138708:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699132.138721:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146241408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699132.138730:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138734:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699132.138735:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138739:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138743:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138746:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138751:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699132.138753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699132.138764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146241536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699132.138774:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138777:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699132.138779:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138782:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138786:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138789:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138795:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699132.138803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146241664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699132.138814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f5187388a80 x1874531146241152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699132.138814:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138823:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699132.138828:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138837:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138841:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699132.138846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146241152/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699132.138848:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138856:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787699132.138865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.138872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699132.138872:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146241792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699132.138880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146241152:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699132.138890:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138894:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699132.138895:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138900:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138903:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138907:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699132.138920:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146241920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699132.138929:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699132.138931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241280:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699132.138933:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699132.138938:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138943:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138946:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.138950:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699132.138954:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699132.138965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699132.138967:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146242048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699132.138977:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699132.138980:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699132.138983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f518738b800 x1874531146241280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699132.138983:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699132.138990:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7031 00000100:00080000:3.0:1787699132.138996:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699132.139002:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146241280/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699132.139008:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699132.139024:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146242176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699132.139029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241280:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699132.139051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699132.139083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f518738bb80 x1874531146241408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146241408/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699132.139126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241536:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699132.139146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f5187388380 x1874531146241536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146241536/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241536:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699132.139189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699132.139206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f518738b480 x1874531146241664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146241664/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146241664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699132.139248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146241792:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699132.139276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f5187389500 x1874531146241792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146241792/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146241792:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699132.139333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146241920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699132.139375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f5187389c00 x1874531146241920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146241920/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146241920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699132.139437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146242048:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699132.139458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f5187388a80 x1874531146242048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146242048/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146242048:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699132.139496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146242176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699132.139512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699132.139521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699132/real 1787699132] req@ffff8f518738aa00 x1874531146242176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699132.139531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146242176/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699132.139538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699132.139541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699132.139543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146242176:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699133.035615:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699134.058593:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699135.085546:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699136.106535:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699137.130513:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699137.258485:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258493:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699137.258497:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699137.258505:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.258516:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699137.258522:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699137.258565:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146242304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699137.258594:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258598:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699137.258600:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699137.258604:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.258608:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699137.258613:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699137.258622:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699137.258636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146242304:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699137.258640:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146242432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699137.258652:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258655:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699137.258657:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699137.258660:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.258663:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699137.258667:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699137.258671:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699137.258682:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146242560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699137.258690:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258693:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699137.258695:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699137.258698:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.258702:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787699137.258707:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699137.258708:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699137.258716:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699137.258740:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146242688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699137.258753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699137.258758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f518738ad80 x1874531146242304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699137.258759:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699137.258763:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699137.258771:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.258778:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699137.258784:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699137.258789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146242304/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699137.258790:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699137.258804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699137.258806:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146242816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699137.258812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.258819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146242304:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699137.258824:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258829:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699137.258831:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699137.258836:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.258840:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699137.258843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242432:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699137.258847:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699137.258854:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787699137.258871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699137.258877:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146242944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699137.258883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f5187388e00 x1874531146242432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699137.258914:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258918:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699137.258919:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699137.258926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146242432/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699137.258934:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:2.0:1787699137.258940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699137.258941:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699137.258946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699137.258950:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699137.258954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242432:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699137.258958:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699137.258962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699137.258974:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146243072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699137.258984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699137.258984:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.258990:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699137.258992:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699137.258998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f518738b100 x1874531146242560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699137.258999:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.259005:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699137.259012:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699137.259017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146242560/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699137.259018:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699137.259029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699137.259034:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146243200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699137.259047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242560:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699137.259048:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699137.259056:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699137.259058:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699137.259059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242688:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699137.259064:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7036 00000100:00080000:3.0:1787699137.259071:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699137.259078:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699137.259082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699137.259090:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699137.259095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f518738a300 x1874531146242688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699137.259107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146242688/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699137.259117:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146243328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699137.259118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.259128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242688:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699137.259138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699137.259155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699137.259164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f5187389f80 x1874531146242816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699137.259175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146242816/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699137.259182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.259187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146242816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699137.259201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146242944:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699137.259214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699137.259222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f5187388000 x1874531146242944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699137.259232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146242944/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699137.259240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.259245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146242944:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699137.259249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146243072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699137.259261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699137.259268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f5187389180 x1874531146243072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699137.259278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146243072/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699137.259291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.259307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146243072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699137.259312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146243200:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699137.259375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699137.259394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f518738a680 x1874531146243200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699137.259405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146243200/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699137.259412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.259418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146243200:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699137.259423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146243328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699137.259441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699137.259451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699137/real 1787699137] req@ffff8f518738ad80 x1874531146243328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699137.259460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146243328/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699137.259467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699137.259471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699137.259474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146243328:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699138.154562:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699139.178530:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699140.202617:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699141.226537:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699142.250513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699142.378535:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378549:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699142.378555:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.378568:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378583:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699142.378592:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699142.378643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146243456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699142.378678:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378682:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699142.378684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.378689:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378693:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699142.378697:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.378709:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699142.378724:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146243584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699142.378736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146243456:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699142.378746:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378753:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699142.378754:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.378759:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378763:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699142.378767:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.378772:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699142.378787:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146243712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699142.378797:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378801:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699142.378803:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.378807:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378812:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699142.378815:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.378821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699142.378835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699142.378836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146243840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699142.378847:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378851:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699142.378853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.378857:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378861:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699142.378865:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.378871:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699142.378884:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146243968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699142.378894:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378897:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699142.378899:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.378903:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378911:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699142.378920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f518738aa00 x1874531146243456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699142.378920:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.378928:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699142.378946:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146244096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699142.378953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146243456/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699142.378965:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.378971:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699142.378972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699142.378973:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699142.378980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699142.378982:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.378988:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699142.378991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146243456:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699142.378995:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.379003:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699142.379015:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146244224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699142.379026:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699142.379029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243584:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699142.379033:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699142.379038:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.379042:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.379046:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699142.379051:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.379056:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699142.379062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699142.379070:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146244352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699142.379079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f5187388a80 x1874531146243584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699142.379083:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699142.379091:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699142.379095:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699142.379100:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7042 00000100:00080000:3.0:1787699142.379105:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699142.379108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146243584/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699142.379114:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699142.379123:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699142.379126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699142.379137:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146244480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699142.379139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243584:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699142.379154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243712:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699142.379179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f5187389c00 x1874531146243712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146243712/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699142.379221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243840:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699142.379251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f5187389500 x1874531146243840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146243840/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243840:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699142.379291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243968:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699142.379308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f518738b480 x1874531146243968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146243968/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146243968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699142.379347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244096:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699142.379404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f5187388380 x1874531146244096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146244096/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244096:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699142.379462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244224:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699142.379478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f518738bb80 x1874531146244224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146244224/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699142.379516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244352:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699142.379543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f518738b800 x1874531146244352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146244352/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244352:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699142.379588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244480:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699142.379607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699142.379617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699142/real 1787699142] req@ffff8f518738aa00 x1874531146244480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699142.379628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146244480/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699142.379634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699142.379637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699142.379640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146244480:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699143.274593:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699144.298610:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699145.322621:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699146.346596:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699147.370573:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699147.498400:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498409:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699147.498411:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498419:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498427:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498432:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699147.498459:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146244608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498489:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498493:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699147.498494:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498499:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498502:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498506:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498515:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699147.498530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146244608:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699147.498531:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146244736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498545:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498547:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699147.498549:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498551:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498554:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498556:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498560:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699147.498569:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146244864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699147.498579:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498581:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498584:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498586:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498590:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699147.498600:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146244992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498611:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699147.498613:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699147.498615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146244736:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699147.498627:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498632:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498636:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699147.498638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146244864:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699147.498643:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699147.498656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146245120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699147.498661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146244992:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699147.498670:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498674:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699147.498675:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498679:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498683:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498687:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498691:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699147.498695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146245120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699147.498704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146245248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498720:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699147.498724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245248:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699147.498725:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699147.498729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498734:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498745:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498750:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498755:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699147.498766:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146245376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498780:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498783:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699147.498784:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498787:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498790:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498793:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498796:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699147.498803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699147.498805:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146245504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699147.498816:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699147.498818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699147.498819:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699147.498821:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7047 00000100:00080000:3.0:1787699147.498823:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699147.498826:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699147.498829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699147.498836:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146245632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699147.498844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699147.498867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245632:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699148.394532:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699149.418538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699150.442563:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787699150.570596:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:0.0:1787699150.570633:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787699150.570650:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570700:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570714:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570724:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570731:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570739:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570746:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570753:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:0.0:1787699150.570760:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:2.0:1787699150.571168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f518738aa00 x1874531146245632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146245632/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245632:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699150.571227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f5187388e00 x1874531146245504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146245504/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245504:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699150.571259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f518738b100 x1874531146245376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146245376/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245376:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699150.571293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f518738a300 x1874531146245248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146245248/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146245248:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699150.571324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f5187389f80 x1874531146245120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146245120/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146245120:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699150.571380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f518738a680 x1874531146244736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146244736/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146244736:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699150.571412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f5187389180 x1874531146244864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146244864/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146244864:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699150.571443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f5187388000 x1874531146244992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146244992/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146244992:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699150.571474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699147/real 1787699150] req@ffff8f518738ad80 x1874531146244608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699150.571487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146244608/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699150.571496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699150.571500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699150.571504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146244608:192.168.201.125@tcp:250:kworker.0 00010000:00010000:1.0:1787699151.466464:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699152.490501:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699152.618453:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618464:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699152.618467:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618477:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618487:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618494:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699152.618528:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146245760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618561:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618565:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699152.618566:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618571:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618575:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618579:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618588:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699152.618600:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146245888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618610:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618614:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699152.618616:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618619:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618622:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618626:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618630:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699152.618645:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146246016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618654:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618657:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699152.618658:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618662:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618665:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618668:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618671:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699152.618678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146246144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618684:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618686:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699152.618688:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618690:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618693:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618695:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618698:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699152.618704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146246272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618710:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618712:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699152.618713:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618715:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618717:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618720:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618722:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699152.618727:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146246400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618734:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618737:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699152.618738:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618741:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618744:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618747:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699152.618759:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146246528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618766:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618769:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699152.618771:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618774:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618777:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618787:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618791:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699152.618802:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146246656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699152.618810:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699152.618813:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699152.618815:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699152.618818:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7052 00000100:00080000:3.0:1787699152.618821:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699152.618824:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699152.618828:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699152.618839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146246784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699152.619843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146245760:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699152.619914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.619954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f518738b800 x1874531146245760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.619970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146245760/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.619980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.619984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.619988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146245760:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699152.619995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146245888:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699152.620015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f518738bb80 x1874531146245888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146245888/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146245888:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699152.620061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146246016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699152.620112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f5187388380 x1874531146246016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146246016/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146246016:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699152.620153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146246144:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699152.620172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f518738b480 x1874531146246144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146246144/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146246144:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699152.620217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146246272:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699152.620235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f5187389500 x1874531146246272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146246272/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146246272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699152.620273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246400:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699152.620292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f5187389c00 x1874531146246400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146246400/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246400:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699152.620348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699152.620455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f5187388a80 x1874531146246528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146246528/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699152.620499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246656:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699152.620518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f518738ad80 x1874531146246656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146246656/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246656:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699152.620556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699152.620573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699152.620585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699152/real 1787699152] req@ffff8f5187388000 x1874531146246784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699152.620596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146246784/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699152.620603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699152.620605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699152.620608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146246784:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699153.514658:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699154.539346:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699155.565465:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699156.586609:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699157.610648:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699157.738475:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738486:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699157.738490:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738499:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738509:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738515:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699157.738548:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146246912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699157.738579:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738582:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699157.738583:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738587:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738590:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738593:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738602:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699157.738613:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146247040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699157.738620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146246912:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699157.738625:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738629:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699157.738631:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738634:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738637:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738641:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738646:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699157.738657:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146247168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699157.738665:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738668:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699157.738670:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738674:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738677:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738682:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699157.738684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738688:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699157.738698:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146247296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699157.738706:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738709:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699157.738710:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738714:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738717:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738720:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699157.738733:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146247424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699157.738741:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738744:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699157.738746:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738751:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00100000:2.0:1787699157.738753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f5187389180 x1874531146246912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699157.738754:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738760:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738767:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699157.738778:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146247552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699157.738782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146246912/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699157.738812:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699157.738813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699157.738818:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699157.738821:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699157.738822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699157.738829:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00100000:2.0:1787699157.738831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146246912:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699157.738833:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738849:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738854:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699157.738857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699157.738867:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146247680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699157.738877:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738880:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699157.738883:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787699157.738884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699157.738888:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738893:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738897:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699157.738901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f518738a680 x1874531146247040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699157.738903:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699157.738917:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146247808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699157.738921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146247040/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699157.738935:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699157.738940:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699157.738942:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699157.738950:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7057 00000100:00080000:3.0:1787699157.738953:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699157.738956:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699157.738960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699157.738968:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146247936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699157.739191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.739195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.739198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247040:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699157.739204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247168:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699157.742189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f5187389f80 x1874531146247168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146247168/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699157.742246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247296:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699157.742261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f518738a300 x1874531146247296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146247296/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247296:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699157.742300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247424:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699157.742323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f518738b100 x1874531146247424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146247424/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146247424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699157.742374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247552:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699157.742390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f5187388e00 x1874531146247552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146247552/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247552:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699157.742447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699157.742462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f518738aa00 x1874531146247680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146247680/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699157.742500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247808:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699157.742513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f5187389180 x1874531146247808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146247808/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247808:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699157.742550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699157.742564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699157.742571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699157/real 1787699157] req@ffff8f5187388000 x1874531146247936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699157.742581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146247936/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699157.742589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699157.742591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699157.742594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146247936:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699158.634792:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699159.658561:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699160.682513:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699161.706539:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699162.732255:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699162.859256:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859264:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699162.859267:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859275:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859283:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859288:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699162.859319:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146248064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859341:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859343:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699162.859344:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859348:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859350:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859374:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859382:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699162.859390:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146248192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859396:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859398:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699162.859399:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859402:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859404:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859405:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859408:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699162.859412:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146248320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859418:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859420:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699162.859421:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859422:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859424:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859426:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859428:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699162.859433:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146248448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859437:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859440:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699162.859441:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859443:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859444:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859446:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859448:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699162.859453:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146248576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859457:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859459:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699162.859459:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859461:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859463:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859464:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859466:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699162.859473:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146248704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859477:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859480:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699162.859481:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859482:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859484:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859485:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859488:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699162.859493:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146248832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859498:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859499:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699162.859500:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859502:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859503:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859505:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859507:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699162.859511:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146248960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699162.859515:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699162.859517:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699162.859517:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699162.859519:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7062 00000100:00080000:1.0:1787699162.859520:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699162.859522:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699162.859524:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699162.859528:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146249088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699162.859627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146248064:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699162.859682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.859737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183644700 x1874531146248064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.859753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146248064/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.859762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.859766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.859770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146248064:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699162.859776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248192:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699162.859940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.859955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183647b80 x1874531146248192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.859967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146248192/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.859975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.859978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.859981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248192:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699162.859987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699162.860007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183646a00 x1874531146248320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146248320/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699162.860048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248448:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699162.860064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183646d80 x1874531146248448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146248448/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248448:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699162.860101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699162.860116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183644e00 x1874531146248576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146248576/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146248576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699162.860157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146248704:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699162.860173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183647800 x1874531146248704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146248704/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146248704:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699162.860211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146248832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699162.860223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183646680 x1874531146248832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146248832/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146248832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699162.860264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146248960:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699162.860276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183644000 x1874531146248960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146248960/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146248960:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699162.860310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146249088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699162.860321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699162.860328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff8f5183645c00 x1874531146249088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699162.860338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146249088/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699162.860345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699162.860347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699162.860351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146249088:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699163.754815:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699164.778969:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699165.802585:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699166.826520:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699167.850533:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699167.979212:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979220:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699167.979224:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979233:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979243:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979250:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699167.979285:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146249216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699167.979311:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979314:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699167.979315:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979320:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979323:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979326:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979335:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699167.979345:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146249344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699167.979379:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979385:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699167.979386:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979390:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979393:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979397:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979401:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699167.979413:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146249472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699167.979420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146249216:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699167.979426:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979430:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699167.979432:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979436:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979439:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979442:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979447:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699167.979458:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146249600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699167.979467:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979471:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699167.979472:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979475:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979479:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979484:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699167.979485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979491:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699167.979503:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146249728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699167.979512:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979515:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699167.979516:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979519:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979523:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979527:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979532:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699167.979533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f518738ad80 x1874531146249216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699167.979545:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146249856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699167.979555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146249216/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699167.979556:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979564:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699167.979568:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699167.979577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.979582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.979586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146249216:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699167.979588:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979596:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699167.979601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249344:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699167.979602:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979611:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699167.979625:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146249984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699167.979633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699167.979638:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979642:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699167.979645:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699167.979649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f5187388a80 x1874531146249344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699167.979652:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979657:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979664:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699167.979670:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699167.979671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146249344/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.979682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699167.979683:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146250112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699167.979687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.979695:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249344:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699167.979702:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699167.979706:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699167.979708:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699167.979712:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7067 00000100:00080000:3.0:1787699167.979715:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699167.979719:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699167.979723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249472:192.168.201.125@tcp:38:kworker.0 00000008:00100000:3.0:1787699167.979724:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699167.979740:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146250240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699167.979750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.979765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f5187389c00 x1874531146249472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.979778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146249472/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.979786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.979789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.979792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249472:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699167.979798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249600:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699167.979829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.979839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f5187389500 x1874531146249600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.979849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146249600/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.979914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.979919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.979923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249600:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699167.979929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699167.979947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.979957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f518738b480 x1874531146249728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.979968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146249728/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.979976:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.979979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.979982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146249728:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699167.979987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146249856:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699167.980001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.980009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f5187388380 x1874531146249856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.980019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146249856/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.980027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.980030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.980033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146249856:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699167.980038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146249984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699167.980051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.980059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f518738bb80 x1874531146249984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.980070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146249984/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.980077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.980080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.980083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146249984:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699167.980088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146250112:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699167.980102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.980109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f518738b800 x1874531146250112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.980119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146250112/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.980126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.980129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.980132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146250112:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699167.980149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146250240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699167.980163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699167.980171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff8f5187388a80 x1874531146250240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699167.980180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146250240/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699167.980188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699167.980190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699167.980193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146250240:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699168.875199:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699169.899257:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699170.922606:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699171.946680:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699172.970551:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699173.098499:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.098512:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699173.098517:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.098530:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.098541:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.098549:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699173.098586:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146250368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699173.098626:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.098630:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699173.098632:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.098640:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.098644:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.098648:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699173.098658:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699173.098670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146250368:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699173.098672:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146250496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699173.098685:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.098689:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699173.098691:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.098694:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.098698:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.098703:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699173.098708:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699173.098737:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146250624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699173.098753:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699173.098758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699173.098769:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699173.098771:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.098777:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.098781:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.098785:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699173.098790:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699173.098804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704e680 x1874531146250368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699173.098805:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146250752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699173.098827:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.098830:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699173.098832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.098835:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.098838:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699173.098839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146250368/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699173.098843:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699173.098850:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787699173.098855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.098860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699173.098861:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146250880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699173.098864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146250368:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787699173.098878:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.098881:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699173.098883:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699173.098887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250496:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699173.098889:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.098895:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.098900:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699173.098905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699173.098909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.098925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704d500 x1874531146250496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699173.098925:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146251008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699173.098955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146250496/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699173.098959:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.098964:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699173.098967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.098973:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:0.0:1787699173.098975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699173.098976:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.098983:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:0.0:1787699173.098984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:2.0:1787699173.098990:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699173.098998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250496:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699173.099007:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146251136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699173.099008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250624:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699173.099023:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.099027:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699173.099029:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699173.099034:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7072 00000800:00000100:0.0:1787699173.099039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699173.099039:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699173.099046:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699173.099051:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699173.099060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704d880 x1874531146250624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699173.099064:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146251264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699173.099092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146250624/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699173.099094:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699173.099101:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699173.099106:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699173.099113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699173.099114:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7072 00000100:00080000:2.0:1787699173.099121:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699173.099122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787699173.099128:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250624:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787699173.099135:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699173.099140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699173.099151:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146251392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699173.099167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704e300 x1874531146250752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699173.099190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146250752/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699173.099198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.099202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699173.099205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250752:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699173.099211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699173.099229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704ea00 x1874531146250880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699173.099252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146250880/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699173.099259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.099261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699173.099264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146250880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699173.099279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251008:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699173.099297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704dc00 x1874531146251008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699173.099312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146251008/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699173.099318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.099320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699173.099321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251008:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699173.099325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699173.099335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704ed80 x1874531146251136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699173.099346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146251136/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699173.099385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.099387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699173.099389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699173.099392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251264:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699173.099404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704f480 x1874531146251264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699173.099420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146251264/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699173.099427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.099430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699173.099433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251264:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699173.099438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699173.099456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699173.099465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699173/real 1787699173] req@ffff8f518704e680 x1874531146251392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699173.099473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146251392/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699173.099477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699173.099479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699173.099482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146251392:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699173.994538:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699175.018553:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699176.042525:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699177.066605:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699178.092909:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699178.218461:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218472:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699178.218475:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218485:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.218495:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.218502:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699178.218536:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146251520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699178.218564:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218568:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699178.218570:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218574:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.218578:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.218582:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699178.218592:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699178.218617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146251520:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699178.218619:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146251648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699178.218668:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218671:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699178.218673:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218677:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.218681:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.218684:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699178.218689:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787699178.218701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699178.218702:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146251776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699178.218712:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218715:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699178.218716:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218720:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.218723:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.218727:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699178.218731:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699178.218742:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146251904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699178.218747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704df80 x1874531146251520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699178.218754:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218763:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699178.218767:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218772:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.218775:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699178.218777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146251520/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699178.218781:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699178.218788:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787699178.218795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.218803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699178.218803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146252032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699178.218811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146251520:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787699178.218820:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218824:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699178.218826:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218830:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.218834:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.218838:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699178.218842:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699178.218849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146251648:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699178.218855:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146252160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699178.218866:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699178.218869:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699178.218871:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218875:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7077 00000800:00000100:0.0:1787699178.218880:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699178.218881:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.218888:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699178.218893:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699178.218899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704c700 x1874531146251648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699178.218907:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146252288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699178.218919:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699178.218924:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146251648/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699178.218924:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699178.218928:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699178.218937:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:0.0:1787699178.218942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699178.218943:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699178.218949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:2.0:1787699178.218953:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699178.218955:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146251648:192.168.201.125@tcp:38:kworker.0 00000008:00100000:2.0:1787699178.218960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699178.218965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146251776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699178.218974:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146252416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699178.218983:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699178.218985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699178.218990:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699178.218993:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699178.218998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704fb80 x1874531146251776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699178.218998:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7077 00000100:00080000:2.0:1787699178.219004:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699178.219011:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146251776/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699178.219017:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787699178.219020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146251776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699178.219034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146251904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699178.219035:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146252544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699178.219046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704d180 x1874531146251904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699178.219063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146251904/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699178.219068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146251904:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699178.219075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146252032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699178.219084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704c000 x1874531146252032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699178.219095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146252032/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699178.219100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146252032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699178.219114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252160:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699178.219123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704c380 x1874531146252160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699178.219134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146252160/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699178.219139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252160:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699178.219145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699178.219153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704ce00 x1874531146252288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699178.219164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146252288/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699178.219172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699178.219178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252416:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699178.219186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704f800 x1874531146252416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699178.219196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146252416/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699178.219200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252416:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699178.219206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699178.219214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699178.219218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699178/real 1787699178] req@ffff8f518704df80 x1874531146252544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699178.219224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146252544/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699178.219229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699178.219230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699178.219232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146252544:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699179.114531:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699180.138567:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699181.162525:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699182.186732:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699183.210592:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699183.338540:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.338580:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699183.338586:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.338600:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.338614:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.338623:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699183.338674:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146252672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699183.338707:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.338710:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699183.338712:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.338715:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.338718:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.338721:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.338731:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699183.338740:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146252800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699183.338747:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.338750:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699183.338751:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.338757:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.338759:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699183.338764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146252672:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699183.338764:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.338769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699183.338779:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146252928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699183.338785:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.338788:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699183.338789:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.338792:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.338794:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.338797:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.338801:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699183.338809:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146253056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699183.338817:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.338821:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699183.338822:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.338826:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.338831:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.338835:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.338839:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787699183.338849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699183.338852:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146253184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699183.338863:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.338868:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699183.338870:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.338874:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.338878:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.338882:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.338899:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699183.338906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f518738ad80 x1874531146252672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699183.338914:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146253312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699183.338935:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699183.338936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146252672/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699183.338945:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699183.338949:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699183.338958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.338964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.338971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146252672:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699183.338974:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00100000:0.0:1787699183.338981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146252800:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699183.338981:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.338988:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.338994:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699183.339009:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146253440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699183.339017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699183.339025:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.339029:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699183.339031:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699183.339035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f5187388000 x1874531146252800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699183.339038:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.339045:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.339053:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146252800/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699183.339061:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787699183.339070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699183.339079:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146253568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699183.339081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146252800:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699183.339101:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699183.339104:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699183.339105:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699183.339109:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7082 00000100:00080000:3.0:1787699183.339113:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699183.339118:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699183.339123:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699183.339124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146252928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699183.339138:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146253696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699183.339168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f5187389180 x1874531146252928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146252928/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146252928:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699183.339217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146253056:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699183.339237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f518738aa00 x1874531146253056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146253056/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146253056:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699183.339281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146253184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699183.339298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f5187388e00 x1874531146253184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146253184/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146253184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699183.339326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253312:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699183.339337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f518738b100 x1874531146253312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146253312/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253312:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699183.339419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699183.339438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f518738a300 x1874531146253440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146253440/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699183.339476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253568:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699183.339490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f5187389f80 x1874531146253568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146253568/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253568:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699183.339527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699183.339557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699183.339569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699183/real 1787699183] req@ffff8f518738a680 x1874531146253696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699183.339580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146253696/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699183.339585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699183.339588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699183.339591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146253696:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699184.234560:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699185.258546:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699186.282578:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699187.306625:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699188.330598:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699188.458505:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.458516:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699188.458520:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458530:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458539:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.458546:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699188.458583:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146253824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699188.458618:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.458623:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699188.458625:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458629:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458632:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.458636:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.458647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699188.458661:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146253952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699188.458669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146253824:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699188.458676:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.458695:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699188.458697:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458702:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458717:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.458722:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.458727:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787699188.458733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699188.458741:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146254080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699188.458752:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.458755:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699188.458757:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458760:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458764:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.458768:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.458772:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699188.458783:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146254208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699188.458792:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699188.458794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f518738a680 x1874531146253824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699188.458800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699188.458803:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458812:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458817:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699188.458824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146253824/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699188.458824:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699188.458838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.458844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787699188.458846:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699188.458848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146253824:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699188.458863:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146254336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699188.458870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146253952:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699188.458882:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.458898:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787699188.458899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699188.458901:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458910:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458914:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699188.458918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f5187389f80 x1874531146253952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699188.458922:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.458938:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699188.458939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146253952/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.458951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699188.458955:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146254464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699188.458958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.458966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146253952:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699188.458975:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699188.458977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146254080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699188.458980:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699188.458985:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.458990:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.458993:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.458997:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.459002:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787699188.459006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699188.459019:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146254592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699188.459024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f518738a300 x1874531146254080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699188.459036:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.459041:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699188.459043:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699188.459045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146254080/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699188.459049:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.459054:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.459058:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.459066:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699188.459079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699188.459080:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146254720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699188.459087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146254080:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699188.459098:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699188.459103:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699188.459105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146254208:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699188.459106:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699188.459116:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7088 00000100:00080000:3.0:1787699188.459121:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699188.459125:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699188.459130:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699188.459136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699188.459143:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146254848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699188.459162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f518738b100 x1874531146254208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699188.459176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146254208/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.459183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.459187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146254208:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699188.459213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146254336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699188.459233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699188.459243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f5187388e00 x1874531146254336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699188.459254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146254336/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.459260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.459271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146254336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699188.459280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254464:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699188.459314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699188.459326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f518738aa00 x1874531146254464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699188.459337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146254464/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.459352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.459380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254464:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699188.459390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699188.459418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699188.459431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f5187389180 x1874531146254592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699188.459442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146254592/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.459450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.459453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699188.459483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254720:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699188.459502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699188.459522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f5187388000 x1874531146254720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699188.459533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146254720/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.459540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.459544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254720:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699188.459552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699188.459579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699188.459590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699188/real 1787699188] req@ffff8f518738ad80 x1874531146254848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699188.459601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146254848/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699188.459609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699188.459612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699188.459615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146254848:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699189.354550:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699190.381691:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699191.402559:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699192.426728:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699193.450580:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699193.578493:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578503:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699193.578507:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578517:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578526:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578533:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699193.578568:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146254976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578606:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578610:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699193.578611:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578616:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578619:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578623:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578631:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699193.578643:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146255104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578651:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578654:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699193.578656:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578659:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578662:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578666:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578669:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699193.578680:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146255232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578688:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578691:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699193.578692:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578696:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578699:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578703:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578707:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699193.578717:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146255360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578728:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699193.578730:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578733:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578736:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578739:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578743:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699193.578764:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146255488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578772:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699193.578777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578780:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578783:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578786:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578790:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699193.578798:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146255616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578811:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578814:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699193.578816:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578819:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578822:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578825:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699193.578848:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146255744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578857:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578860:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699193.578861:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578865:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578867:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578871:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578875:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699193.578885:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146255872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699193.578893:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699193.578896:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699193.578898:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699193.578901:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7093 00000100:00080000:3.0:1787699193.578904:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699193.578907:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699193.578911:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699193.578920:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146256000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699193.579009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146254976:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699193.579074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f5187388a80 x1874531146254976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146254976/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146254976:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699193.579152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255104:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699193.579478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f518738b800 x1874531146255104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146255104/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255104:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699193.579524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699193.579542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f518738bb80 x1874531146255232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146255232/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699193.579582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255360:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699193.579598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f5187388380 x1874531146255360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146255360/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255360:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699193.579634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699193.579648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f518738b480 x1874531146255488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146255488/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146255488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699193.579683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146255616:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699193.579697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f5187389500 x1874531146255616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146255616/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146255616:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699193.579734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146255744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699193.579748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f5187389c00 x1874531146255744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146255744/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146255744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699193.579784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146255872:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699193.579798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.579806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f518738ad80 x1874531146255872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.579816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146255872/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.579828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.579830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.579833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146255872:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699193.580015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146256000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699193.580035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699193.580046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699193/real 1787699193] req@ffff8f5187388000 x1874531146256000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699193.580056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146256000/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699193.580116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699193.580119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699193.580123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146256000:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699194.475402:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699195.498614:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699196.522519:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699197.546670:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699198.571740:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699198.699392:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699404:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699198.699408:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699418:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699428:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699435:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699198.699474:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146256128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699507:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699511:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699198.699513:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699517:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699521:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699525:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699534:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699198.699546:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146256256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699555:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699558:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699198.699560:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699563:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699566:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699568:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699572:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699198.699581:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146256384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699589:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699592:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699198.699594:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699597:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699600:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699603:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699607:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699198.699617:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146256512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699624:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699627:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699198.699629:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699632:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699635:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699639:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699643:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699198.699654:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146256640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699674:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699677:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699198.699678:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699682:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699685:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699689:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699694:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699198.699705:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146256768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699712:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699715:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699198.699716:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699719:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699723:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699726:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699730:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699198.699739:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146256896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699760:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699763:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699198.699765:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699768:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699771:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699774:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699779:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699198.699789:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146257024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699198.699797:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699198.699800:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699198.699802:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699198.699806:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7098 00000100:00080000:1.0:1787699198.699809:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699198.699813:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699198.699817:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699198.699956:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146257152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699198.700392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146256128:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699198.700453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.700506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183644380 x1874531146256128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.700522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146256128/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.700531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.700535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.700538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146256128:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699198.700546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256256:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699198.700567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.700579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183647480 x1874531146256256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.700590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146256256/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.700597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.700600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.700604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256256:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699198.700609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699198.700627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.700638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183647100 x1874531146256384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.700648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146256384/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.700656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.700659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.700662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699198.700667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256512:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699198.700687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.700697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183645180 x1874531146256512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.700709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146256512/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.700716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.700719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.700726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256512:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699198.700731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699198.703394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.703419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183645c00 x1874531146256640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.703434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146256640/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.703444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.703447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.703452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146256640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699198.703459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146256768:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699198.703480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.703490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183644000 x1874531146256768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.703502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146256768/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.703515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.703518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.703521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146256768:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699198.703527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146256896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699198.703546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.703557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183646680 x1874531146256896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.703567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146256896/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.703574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.703577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.703580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146256896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699198.703586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146257024:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699198.703606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.703617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183647800 x1874531146257024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.703628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146257024/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.703635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.703638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.703641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146257024:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699198.703647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146257152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699198.703666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699198.703676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699198/real 1787699198] req@ffff8f5183644e00 x1874531146257152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699198.703687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146257152/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699198.703694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699198.703697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699198.703700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146257152:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699199.594646:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699200.621127:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699201.642751:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699202.666619:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699203.690689:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699203.818515:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818525:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699203.818529:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818539:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818550:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818556:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699203.818734:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146257280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818765:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699203.818767:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818771:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818775:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818779:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.818788:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699203.818800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146257408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818810:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818813:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699203.818814:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818818:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818821:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818825:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.818829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699203.818839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146257536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818851:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699203.818853:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818857:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818860:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818864:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.818868:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699203.818876:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146257664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818885:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818889:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699203.818890:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818894:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818897:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818900:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.818905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699203.818915:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146257792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818924:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818927:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699203.818929:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818933:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818935:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818939:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.818944:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699203.818953:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146257920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818963:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.818966:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699203.818968:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.818971:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.818974:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.818977:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.818981:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699203.818990:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146258048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.818998:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.819001:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699203.819003:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.819006:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.819010:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.819013:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.819017:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699203.819029:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146258176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699203.819037:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699203.819041:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699203.819042:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699203.819045:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7103 00000100:00080000:0.0:1787699203.819048:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699203.819051:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699203.819055:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699203.819063:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146258304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699203.819141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146257280:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699203.819224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.819270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace91880 x1874531146257280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.819286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146257280/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.819296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.819300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.819303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146257280:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699203.819310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257408:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699203.819327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.819337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace93800 x1874531146257408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.819348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146257408/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.819380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.819384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.819387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257408:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699203.819393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699203.819411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.819422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace90380 x1874531146257536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.819432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146257536/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.819439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.819442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.819445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699203.819450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257664:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699203.819467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.819476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace92680 x1874531146257664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.819487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146257664/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.819494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.819497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.819500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257664:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699203.819505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699203.819518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.819525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace91500 x1874531146257792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.819535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146257792/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.819543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.819545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.819548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146257792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699203.819552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146257920:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699203.819565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.819572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace93480 x1874531146257920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.820301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146257920/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.820313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.820316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.820321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146257920:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699203.820327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146258048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699203.820368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.820383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace92d80 x1874531146258048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.820395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146258048/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.820403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.820405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.820409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146258048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699203.820415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146258176:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699203.820431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.820439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace90e00 x1874531146258176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.820450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146258176/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.820457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.820460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.820463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146258176:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699203.820468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146258304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699203.820480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699203.820488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699203/real 1787699203] req@ffff8f51ace93b80 x1874531146258304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699203.820498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146258304/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699203.820506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699203.820508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699203.820511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146258304:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699204.714632:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699205.738565:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699206.763656:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699207.787230:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699208.815766:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699208.938721:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.938732:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699208.938736:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.938746:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.938757:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.938764:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699208.938800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146258432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.938831:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.938835:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699208.938837:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.938842:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.938846:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.938850:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.938859:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699208.938872:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146258560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.938882:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.938887:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699208.938889:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.938892:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.938896:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.938899:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.938904:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699208.938915:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146258688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.938923:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.938926:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699208.938928:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.938932:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.938934:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.938938:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.938942:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699208.938952:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146258816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.938960:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.938963:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699208.938964:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.938968:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.938971:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.938974:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.938978:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699208.938987:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146258944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.938995:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.938998:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699208.939000:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.939003:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.939006:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.939009:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.939013:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699208.939089:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146259072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.939098:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.939101:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699208.939102:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.939105:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.939123:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.939128:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.939133:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699208.939144:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146259200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.939152:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.939155:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699208.939157:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.939160:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.939163:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.939166:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.939170:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699208.939180:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146259328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699208.939189:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699208.939191:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699208.939193:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699208.939196:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7108 00000100:00080000:3.0:1787699208.939199:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699208.939202:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699208.939205:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699208.939227:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146259456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699208.940315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146258432:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787699208.941430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f5187389180 x1874531146258432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146258432/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146258432:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699208.941571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258560:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699208.941595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f518738aa00 x1874531146258560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146258560/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258560:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699208.941636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258688:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699208.941652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f5187388e00 x1874531146258688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146258688/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699208.941687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258816:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699208.941703:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f518738b100 x1874531146258816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146258816/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258816:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699208.941739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258944:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699208.941755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f518738a300 x1874531146258944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146258944/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146258944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699208.941792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259072:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699208.941810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f5187389f80 x1874531146259072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146259072/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259072:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699208.941851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259200:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699208.941869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f518738a680 x1874531146259200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146259200/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941899:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699208.941904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259328:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699208.941918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f5187388000 x1874531146259328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146259328/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.941944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.941947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.941951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259328:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699208.941957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259456:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699208.941974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699208.941983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff8f518738ad80 x1874531146259456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.941994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146259456/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.942001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.942004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699208.942007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146259456:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699209.842146:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699210.858577:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699211.882668:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699212.906509:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699213.930988:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699214.058472:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058484:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699214.058488:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058498:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058510:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058516:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699214.058551:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146259584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058583:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058587:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699214.058589:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058596:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058601:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058610:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699214.058621:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146259712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058630:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058633:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699214.058634:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058637:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058640:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058643:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058647:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699214.058656:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146259840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058664:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058667:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699214.058668:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058671:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058674:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058678:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058682:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699214.058691:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146259968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058699:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058702:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699214.058704:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058707:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058710:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058713:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058717:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699214.058726:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146260096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058734:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058736:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699214.058738:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058741:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058744:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058747:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058751:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699214.058760:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146260224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058768:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058771:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699214.058772:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058775:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058778:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058781:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058785:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699214.058794:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146260352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058802:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699214.058806:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058809:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058812:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058815:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058819:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699214.058828:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146260480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699214.058836:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699214.058839:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699214.058840:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699214.058843:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7113 00000100:00080000:3.0:1787699214.058845:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699214.058849:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699214.058853:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699214.058861:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146260608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699214.060101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146259584:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699214.060192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146259712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699214.060215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146259840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699214.060230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146259968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699214.060246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146260096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699214.060260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699214.060275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699214.060290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699214.060303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260608:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699214.954572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699215.978662:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699217.002612:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787699217.130634:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:1.0:1787699217.130680:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787699217.130693:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130737:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130749:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130757:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130764:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130772:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130779:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130786:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:1.0:1787699217.130793:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:0.0:1787699217.131034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f518738ad80 x1874531146259584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146259584/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146259584:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699217.131089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f5187388000 x1874531146259712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146259712/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146259712:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699217.131130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f518738a680 x1874531146259840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146259840/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146259840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699217.131162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f5187389f80 x1874531146259968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146259968/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146259968:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699217.131193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f518738a300 x1874531146260096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146260096/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146260096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699217.131226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f518738b100 x1874531146260224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146260224/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260224:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699217.131258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f5187388e00 x1874531146260352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146260352/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699217.131288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f518738aa00 x1874531146260480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146260480/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260480:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699217.131320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699214/real 1787699217] req@ffff8f5187389180 x1874531146260608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699217.131332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146260608/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699217.131339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699217.131342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699217.131345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146260608:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699218.026663:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699219.071817:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699219.178898:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.178909:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699219.178913:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.178922:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.178932:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.179009:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699219.179042:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146260736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699219.179159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146260736:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787699219.179215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699219.179738:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.179744:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699219.179746:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.179751:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.179756:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.179761:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.179771:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699219.179787:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146260864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.180008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183646d80 x1874531146260736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.180028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146260736/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.180038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.180041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.180045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146260736:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699219.180070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146260864:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699219.180101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699219.180115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183646a00 x1874531146260864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.180126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146260864/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.180140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.180143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.180147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146260864:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699219.180261:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.180265:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699219.180267:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.180271:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.180276:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.180279:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.180284:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699219.180296:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146260992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.180337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146260992:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699219.180372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699219.180405:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.180408:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699219.180411:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.180415:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.180418:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.180422:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.180428:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699219.180438:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146261120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.180473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183647b80 x1874531146260992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.180484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146260992/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.180492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.180495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.180498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146260992:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699219.180509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146261120:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699219.180527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699219.180537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183644700 x1874531146261120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.180548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146261120/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.180556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.180559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.180562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146261120:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787699219.180642:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.180646:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699219.180648:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.180652:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.180656:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.180659:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.180665:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699219.180678:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146261248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.180720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146261248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699219.180744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699219.180767:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.180771:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699219.180773:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.180777:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.180781:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.180785:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.180790:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699219.180803:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146261376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.180849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183644700 x1874531146261248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.180862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146261248/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.180870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.180873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.180876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146261248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699219.180887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261376:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699219.180906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699219.180916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183647b80 x1874531146261376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.180927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146261376/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.180945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.180949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.180952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261376:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699219.181037:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.181041:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699219.181044:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.181048:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.181052:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.181056:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.181062:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699219.181074:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146261504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.181117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699219.181139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699219.181160:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.181164:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699219.181166:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.181170:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.181173:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.181178:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.181182:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699219.181194:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146261632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.181230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183646a00 x1874531146261504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.181243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146261504/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.181251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.181254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.181257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699219.181270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261632:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699219.181291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699219.181302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183646d80 x1874531146261632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.181313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146261632/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.181320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.181323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.181326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261632:192.168.201.125@tcp:8:kworker.0 00000100:00080000:1.0:1787699219.181429:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699219.181433:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699219.181435:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699219.181440:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7118 00000100:00080000:1.0:1787699219.181444:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699219.181448:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699219.181453:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699219.181467:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146261760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699219.181511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699219.181538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699219.181629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699219/real 1787699219] req@ffff8f5183644e00 x1874531146261760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699219.181641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146261760/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699219.181649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699219.181652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699219.181655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146261760:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699220.074621:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699221.098606:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699222.123223:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699223.146545:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699224.170731:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699224.298553:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.298564:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699224.298569:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.298579:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.298590:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.298598:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699224.298638:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146261888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699224.298728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146261888:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699224.298807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699224.298840:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.298844:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699224.298846:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.298867:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.298872:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.298876:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.298886:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699224.298901:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146262016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699224.299274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f5187389c00 x1874531146261888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.299307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146261888/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.299318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.299323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.299327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146261888:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699224.299395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262016:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699224.299424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.299437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f5187389500 x1874531146262016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.299450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146262016/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.299459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.299462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.299466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262016:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699224.299635:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.299639:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699224.299641:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.299646:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.299650:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.299654:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.299660:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699224.299674:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146262144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699224.299735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699224.299758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699224.299779:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.299783:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699224.299785:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.299790:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.299794:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.299798:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.299803:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699224.299818:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146262272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699224.299867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f518738b480 x1874531146262144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.299880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146262144/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.299888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.299891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.299895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699224.299906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262272:192.168.201.125@tcp:8:kworker.0 00000100:00080000:3.0:1787699224.299982:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.299987:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699224.299989:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.300139:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.300144:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.300149:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.300154:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699224.300166:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146262400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699224.300176:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.300180:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699224.300182:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.300186:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.300190:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.300194:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.300198:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699224.300210:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146262528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699224.300219:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.300224:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699224.300226:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.300230:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.300234:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.300238:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.300243:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699224.300254:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146262656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699224.300263:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.300267:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699224.300269:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.300272:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.300275:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.300279:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.300283:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699224.300292:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146262784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699224.300319:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699224.300322:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699224.300324:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699224.300328:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7123 00000100:00080000:3.0:1787699224.300331:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699224.300335:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699224.300340:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699224.300350:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146262912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699224.300458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.300472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f5187388380 x1874531146262272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.300484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146262272/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.300492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.300495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.300499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262272:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699224.300512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699224.300532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.300543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f518738bb80 x1874531146262400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.300556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146262400/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.300564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.300567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.300570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146262400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699224.300576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262528:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699224.300595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.300606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f518738b800 x1874531146262528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.300617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146262528/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.300626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.300628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.300631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262528:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699224.300637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699224.300655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.300666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f5187388a80 x1874531146262656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.300676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146262656/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.300690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.300693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.300696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699224.300702:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262784:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699224.300719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.300729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f518738b480 x1874531146262784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.300742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146262784/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.300750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.300753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.300757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262784:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699224.300762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699224.300781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699224.300791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699224/real 1787699224] req@ffff8f5187389500 x1874531146262912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699224.300802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146262912/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699224.300810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699224.300812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699224.300816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146262912:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699225.194651:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699226.218607:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699227.243533:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699228.266574:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699229.290669:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699229.418488:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418500:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699229.418505:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418515:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418527:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418533:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699229.418570:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146263040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699229.418601:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418605:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699229.418607:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418645:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418649:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418653:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699229.418664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146263040:192.168.201.125@tcp:250:kworker.0 00000008:00100000:3.0:1787699229.418674:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699229.418687:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146263168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699229.418696:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418699:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699229.418700:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418704:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418707:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418710:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699229.418714:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699229.418727:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146263296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699229.418734:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418737:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699229.418739:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418741:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418743:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787699229.418747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699229.418747:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699229.418751:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699229.418758:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146263424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699229.418778:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418781:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699229.418782:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418786:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418789:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418792:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699229.418796:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699229.418799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f5187389c00 x1874531146263040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699229.418804:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146263552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699229.418812:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699229.418815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146263040/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699229.418815:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699229.418817:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699229.418824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.418828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699229.418831:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00100000:2.0:1787699229.418832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146263040:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699229.418833:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418837:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699229.418841:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699229.418850:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146263680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699229.418853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263168:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699229.418859:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418862:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699229.418864:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418867:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418869:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418872:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000800:00000100:2.0:1787699229.418874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000008:00100000:3.0:1787699229.418877:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699229.418886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f5187389180 x1874531146263168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699229.418886:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146263808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699229.418896:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699229.418898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146263168/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699229.418899:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699229.418900:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699229.418906:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.418909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.418912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263168:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699229.418918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263296:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699229.418923:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418926:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699229.418929:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699229.418934:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787699229.418935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699229.418943:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146263936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699229.418946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f518738aa00 x1874531146263296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699229.418952:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699229.418955:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699229.418957:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146263296/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699229.418957:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699229.418960:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7129 00000100:00080000:3.0:1787699229.418963:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699229.418964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699229.418966:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:2.0:1787699229.418967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000008:00100000:3.0:1787699229.418970:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699229.418971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699229.418976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699229.418978:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146264064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787699229.418992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699229.419080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f5187388e00 x1874531146263424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699229.419093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146263424/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699229.419101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.419104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.419106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263424:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699229.419110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699229.419127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699229.419138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f518738b100 x1874531146263552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699229.419147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146263552/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699229.419154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.419157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.419160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146263552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699229.419175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146263680:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699229.419214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699229.419226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f518738a300 x1874531146263680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699229.419237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146263680/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699229.419244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.419247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.419251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146263680:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699229.419256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146263808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699229.419275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699229.419286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f5187389f80 x1874531146263808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699229.419345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146263808/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699229.419352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.419380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.419385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146263808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699229.419390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146263936:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699229.419415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699229.419440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f518738a680 x1874531146263936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699229.419452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146263936/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699229.419459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.419462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.419478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146263936:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699229.419484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146264064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699229.419501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699229.419510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699229/real 1787699229] req@ffff8f5187388000 x1874531146264064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699229.419522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146264064/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699229.419528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699229.419530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699229.419533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146264064:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699230.315451:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699231.351760:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699232.363237:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699233.386898:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699234.411413:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699234.538443:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.538455:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699234.538458:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.538470:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.538481:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.538488:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699234.538527:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146264192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699234.538647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146264192:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699234.538724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699234.539735:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.539741:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699234.539743:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.539749:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.539755:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.539760:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.539769:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699234.539801:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146264320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540422:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540427:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699234.540429:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540434:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540439:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540443:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540449:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699234.540463:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146264448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540480:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540483:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699234.540485:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540488:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540491:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540494:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540499:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699234.540509:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146264576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540517:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540520:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699234.540521:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540525:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540527:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540530:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540535:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699234.540544:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146264704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540552:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540555:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699234.540557:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540560:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540563:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540566:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540570:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699234.540577:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146264832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540585:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540588:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699234.540590:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540593:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540595:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540598:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540601:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699234.540608:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146264960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540615:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540618:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699234.540619:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540798:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540803:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540807:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540811:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699234.540823:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146265088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699234.540838:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699234.540842:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699234.540843:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699234.540847:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7134 00000100:00080000:3.0:1787699234.540850:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699234.540853:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699234.540858:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699234.540867:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146265216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699234.540964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f518738ad80 x1874531146264192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.540985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146264192/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.540994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146264192:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699234.541031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264320:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699234.541073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f5187388000 x1874531146264320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146264320/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264320:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699234.541118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699234.541136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f518738a680 x1874531146264448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146264448/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699234.541179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264576:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699234.541198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f5187389f80 x1874531146264576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146264576/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264576:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699234.541238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699234.541257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f518738a300 x1874531146264704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146264704/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146264704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699234.541294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146264832:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699234.541313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f518738b100 x1874531146264832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146264832/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146264832:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699234.541373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146264960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699234.541393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f5187388e00 x1874531146264960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146264960/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146264960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699234.541435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146265088:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699234.541453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f518738aa00 x1874531146265088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146265088/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146265088:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699234.541492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146265216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699234.541509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699234.541520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699234/real 1787699234] req@ffff8f5187389180 x1874531146265216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699234.541531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146265216/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699234.541538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699234.541540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699234.541543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146265216:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699235.434569:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699236.459247:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699237.482670:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699238.509692:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699239.530546:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699239.658530:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.658541:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699239.658545:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.658554:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.658566:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.658573:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699239.658684:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146265344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.658751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146265344:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699239.658814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699239.658844:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.658848:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699239.658849:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.658854:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.658858:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.658863:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.658873:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699239.658888:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146265472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.658976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace92300 x1874531146265344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.658993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146265344/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.659016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.659020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.659024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146265344:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699239.659069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265472:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699239.659093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699239.659106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace93100 x1874531146265472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.659118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146265472/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.659126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.659129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.659132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265472:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699239.659290:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.659295:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699239.659297:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.659301:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.659441:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.659446:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.659451:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699239.659465:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146265600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.659521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699239.659546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699239.659568:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.659571:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699239.659573:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.659577:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.659581:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.659585:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.659602:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699239.659615:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146265728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.659704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace90700 x1874531146265600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.659717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146265600/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.659725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.659728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.659732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699239.659745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265728:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699239.659766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699239.659777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace90000 x1874531146265728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.659788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146265728/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.659795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.659798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.659801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265728:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787699239.659886:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.659890:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699239.659892:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.659896:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.659900:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.659904:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.659909:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699239.659921:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146265856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.659967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699239.659987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699239.660014:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.660017:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699239.660020:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.660023:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.660027:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.660031:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.660036:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699239.660047:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146265984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.660125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace91180 x1874531146265856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.660137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146265856/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.660145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.660148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.660151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146265856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699239.660163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146265984:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699239.660182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699239.660192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace91c00 x1874531146265984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.660202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146265984/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.660210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.660213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.660216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146265984:192.168.201.125@tcp:38:kworker.0 00000100:00080000:0.0:1787699239.660401:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.660406:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699239.660407:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.660411:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.660415:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.660420:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.660425:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699239.660437:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146266112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.660478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146266112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699239.660565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:0.0:1787699239.660585:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.660599:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699239.660601:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.660605:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.660609:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.660613:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.660618:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699239.660630:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146266240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.660694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace91c00 x1874531146266112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.660706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146266112/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.660714:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.660717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.660720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146266112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699239.660731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146266240:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699239.660750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699239.660761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace91180 x1874531146266240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.660772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146266240/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.660786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.660789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.660792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146266240:192.168.201.125@tcp:8:kworker.0 00000100:00080000:0.0:1787699239.660971:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699239.660975:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699239.660977:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699239.660981:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7139 00000100:00080000:0.0:1787699239.660985:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699239.660989:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699239.660993:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699239.661081:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146266368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699239.661149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146266368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699239.661173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699239.661184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699239/real 1787699239] req@ffff8f51ace90000 x1874531146266368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699239.661196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146266368/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699239.661204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699239.661207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699239.661210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146266368:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699240.554812:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699241.579993:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699242.602712:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699243.626708:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699244.650685:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699244.778725:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.778735:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699244.778739:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.778748:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.778758:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.778766:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699244.778803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146266496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699244.779005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146266496:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699244.779130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699244.779160:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779204:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699244.779206:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779211:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779216:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779331:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779341:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699244.779380:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146266624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699244.779477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f5187389c00 x1874531146266496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.779495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146266496/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699244.779518:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779522:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699244.779524:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779528:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779532:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779535:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779540:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699244.779551:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146266752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699244.779559:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779562:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699244.779564:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779567:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779768:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779774:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779780:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699244.779792:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146266880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699244.779802:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779804:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699244.779807:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779811:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779815:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779818:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779823:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699244.779832:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146267008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699244.779840:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779842:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699244.779843:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779847:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779850:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779853:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779857:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699244.779865:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146267136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699244.779872:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779875:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699244.779876:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779879:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779882:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779885:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779889:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699244.779898:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146267264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699244.779906:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779909:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699244.779910:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779913:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:3.0:1787699244.779916:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.779919:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.779924:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699244.779934:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146267392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699244.779943:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699244.779945:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699244.779946:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699244.779949:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7144 00000100:00080000:2.0:1787699244.779978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.779982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.779986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146266496:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699244.780019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146266624:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699244.780051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.780065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f5187389500 x1874531146266624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.780078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146266624/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.780085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.780088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.780091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146266624:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699244.780097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146266752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699244.780115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.780125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f518738b480 x1874531146266752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.780136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146266752/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.780144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.780146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.780149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146266752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699244.780155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146266880:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699244.780174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.780184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f5187388a80 x1874531146266880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.780196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146266880/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.780203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.780206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.780209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146266880:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699244.780214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146267008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699244.780232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.780242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f518738b800 x1874531146267008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.780253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146267008/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.780260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.780262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.780265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146267008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699244.780270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267136:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699244.780286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.780297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f518738bb80 x1874531146267136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.780308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146267136/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.780322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.780325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.780328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267136:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699244.780333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699244.780352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.780385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f5187388380 x1874531146267264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.780395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146267264/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.780402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.780405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699244.780596:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699244.780601:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699244.780607:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699244.780620:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146267520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699244.780923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699244.780930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267392:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699244.781087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.781102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f5187389180 x1874531146267392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.781114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146267392/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.781122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.781126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.781129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267392:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699244.781156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699244.781177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699244.781187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699244/real 1787699244] req@ffff8f518738aa00 x1874531146267520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699244.781199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146267520/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699244.781206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699244.781209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699244.781212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146267520:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699245.674717:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699246.701698:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699247.722668:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699248.746880:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699249.770921:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699249.898575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898589:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699249.898595:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898608:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898621:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898629:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699249.898678:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146267648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898716:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898721:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699249.898722:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898728:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898731:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898736:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.898748:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699249.898762:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146267776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898772:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699249.898777:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898781:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898784:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898788:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.898793:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699249.898803:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146267904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898819:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898822:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699249.898824:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898828:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898831:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898835:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.898840:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699249.898852:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146268032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898861:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898865:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699249.898866:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898870:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898873:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898876:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.898881:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699249.898890:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146268160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898899:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898902:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699249.898904:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898908:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898912:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898915:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.898919:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699249.898930:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146268288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898939:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898943:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699249.898944:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898948:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898951:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898955:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.898960:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699249.898971:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146268416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.898980:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.898984:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699249.898986:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.898989:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.898993:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.898997:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.899001:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699249.899012:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146268544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699249.899027:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.899030:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699249.899032:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.899036:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7149 00000100:00080000:3.0:1787699249.899039:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699249.899043:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699249.899048:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699249.899057:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146268672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699249.899610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146267648:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699249.899692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.899762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f5187388e00 x1874531146267648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.899784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146267648/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.899795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.899800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.899804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146267648:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699249.899813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146267776:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699249.899851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.899866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f518738b100 x1874531146267776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.899879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146267776/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.899887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.899891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.899894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146267776:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699249.899901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146267904:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699249.899921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.899933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f518738a300 x1874531146267904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.899946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146267904/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.899953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.899956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.899960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146267904:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699249.899966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146268032:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699249.899987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.899998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f5187389f80 x1874531146268032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.900010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146268032/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.900018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.900022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.900025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146268032:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699249.900030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146268160:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699249.900051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.900063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f518738a680 x1874531146268160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.900074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146268160/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.900082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.900086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.900089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146268160:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699249.900095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268288:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699249.900117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.900129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f5187388000 x1874531146268288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.900140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146268288/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.900149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.900152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.900157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268288:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699249.900163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268416:192.168.201.125@tcp:38:kworker.0 00000800:00000100:2.0:1787699249.900182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.900193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f518738ad80 x1874531146268416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.900205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146268416/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.900213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.900216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.900220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268416:192.168.201.125@tcp:38:kworker.0 00000100:00100000:2.0:1787699249.900226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268544:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699249.900245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.900256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f518738aa00 x1874531146268544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.900268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146268544/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.900276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.900279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.900282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268544:192.168.201.125@tcp:8:kworker.0 00000100:00100000:2.0:1787699249.900288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268672:192.168.201.125@tcp:8:kworker.0 00000800:00000100:2.0:1787699249.900321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699249.900332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff8f5187389180 x1874531146268672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699249.900344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146268672/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699249.901497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699249.901504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:2.0:1787699249.901508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146268672:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699250.794677:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699251.821092:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699252.843257:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699253.868388:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699254.890690:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699255.018575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.018586:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699255.018591:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.018601:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.018613:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.018620:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699255.018656:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146268800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699255.018730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146268800:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787699255.018798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699255.018826:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.018830:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699255.018832:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.018837:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.018840:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.018844:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.018852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699255.018864:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146268928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699255.022117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704e680 x1874531146268800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.022150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146268800/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.022162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.022166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.022170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146268800:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699255.022201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146268928:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699255.022259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.022276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704f480 x1874531146268928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.022286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146268928/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.022293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.022296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.022298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146268928:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699255.022396:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022401:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699255.022403:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022408:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022414:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022419:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022429:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699255.022455:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ed80 x1874531146269056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699255.022466:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022469:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699255.022471:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022474:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022477:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022480:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022485:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699255.022494:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704dc00 x1874531146269184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699255.022502:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022505:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699255.022507:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022510:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022514:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022517:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022521:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699255.022531:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ea00 x1874531146269312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699255.022539:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022542:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699255.022544:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022547:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022550:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022553:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022557:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699255.022567:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e300 x1874531146269440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699255.022575:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022578:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699255.022580:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022583:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022586:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022589:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022593:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699255.022603:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d880 x1874531146269568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699255.022616:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022619:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699255.022620:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022623:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022626:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022630:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022634:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699255.022644:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d500 x1874531146269696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699255.022653:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699255.022655:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699255.022657:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699255.022660:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7154 00000100:00080000:2.0:1787699255.022664:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699255.022667:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699255.022671:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699255.022679:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f480 x1874531146269824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699255.022859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146269056:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699255.022890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.022904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704ed80 x1874531146269056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.022917:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ed80 x1874531146269056/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.022925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.022928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146269056:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699255.023091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146269184:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699255.023113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.023124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704dc00 x1874531146269184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.023135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704dc00 x1874531146269184/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.023148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.023152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146269184:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699255.023162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146269312:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699255.023179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.023189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704ea00 x1874531146269312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.023200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ea00 x1874531146269312/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.023208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.023211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146269312:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699255.023219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269440:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699255.023236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.023245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704e300 x1874531146269440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.023256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e300 x1874531146269440/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.023263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.023266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269440:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699255.023275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269568:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699255.023291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.023300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704d880 x1874531146269568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.023386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d880 x1874531146269568/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.023393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.023396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269568:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699255.023406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269696:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699255.023425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.023436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704d500 x1874531146269696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.023447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d500 x1874531146269696/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.023454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.023457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269696:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699255.023465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269824:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699255.023483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699255.023493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699255/real 1787699255] req@ffff8f518704f480 x1874531146269824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699255.023503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f480 x1874531146269824/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699255.023511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699255.023514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699255.023517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146269824:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699255.916069:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699256.940599:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699257.964329:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699258.986567:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699260.011478:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699260.140620:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140630:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699260.140634:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140644:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140655:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140664:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699260.140704:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146269952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.140770:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140775:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699260.140776:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140781:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140786:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140790:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.140799:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699260.140812:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146270080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.140821:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140824:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699260.140826:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140829:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140832:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140836:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.140840:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699260.140851:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146270208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.140860:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140863:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699260.140865:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140868:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140871:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140875:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.140879:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699260.140889:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146270336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.140897:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140900:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699260.140901:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140904:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140906:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140909:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.140913:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699260.140922:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146270464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.140930:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140933:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699260.140935:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140938:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140941:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140945:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.140949:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699260.140961:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146270592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.140969:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.140972:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699260.140974:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.140978:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.140993:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.140996:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.141000:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699260.141009:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146270720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.141018:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.141021:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699260.141022:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.141026:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.141029:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.141033:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.141036:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699260.141045:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146270848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699260.141054:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699260.141056:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699260.141058:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699260.141062:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7159 00000100:00080000:0.0:1787699260.141065:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699260.141068:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699260.141072:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699260.141080:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146270976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699260.142227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146269952:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787699260.142316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace90700 x1874531146269952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146269952/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146269952:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699260.142457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270080:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699260.142482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace93100 x1874531146270080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146270080/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270080:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699260.142532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270208:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699260.142552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace92300 x1874531146270208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146270208/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270208:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699260.142595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270336:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699260.142616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace93b80 x1874531146270336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146270336/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270336:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699260.142657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270464:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699260.142675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace90e00 x1874531146270464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142695:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146270464/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142702:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146270464:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699260.142716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270592:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699260.142734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace92d80 x1874531146270592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146270592/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270592:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699260.142773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270720:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699260.142790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace93480 x1874531146270720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146270720/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142815:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270720:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699260.142825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270848:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699260.142843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace91500 x1874531146270848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146270848/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270848:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699260.142893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270976:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699260.142914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699260.142926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699260/real 1787699260] req@ffff8f51ace92680 x1874531146270976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699260.142936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146270976/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699260.142943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699260.142946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699260.142949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146270976:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699261.035296:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699262.058556:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699263.082603:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699264.107522:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699265.131310:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699265.258445:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258463:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699265.258469:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258482:0:75078:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258498:0:75078:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258509:0:75078:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699265.258568:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146271104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258601:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258605:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699265.258608:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258613:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258616:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258620:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258631:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699265.258645:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146271232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258655:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258658:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699265.258660:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258664:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258667:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258671:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258675:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699265.258685:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146271360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258706:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258709:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699265.258711:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258716:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258719:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258722:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258727:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699265.258746:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146271488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258755:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258759:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699265.258761:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258765:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258768:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258771:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258776:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699265.258786:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146271616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258794:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258798:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699265.258799:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258804:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258807:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258811:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258814:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699265.258824:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146271744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258832:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258835:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699265.258837:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258840:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258843:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258846:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258850:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699265.258859:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146271872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258868:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258871:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699265.258873:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258876:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258879:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258882:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258886:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699265.258895:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146272000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699265.258903:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699265.258906:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699265.258908:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699265.258911:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7164 00000100:00080000:0.0:1787699265.258914:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699265.258918:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699265.258922:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699265.258931:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146272128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699265.260983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146271104:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787699265.261084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.261141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace90380 x1874531146271104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.261172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146271104/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.261184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.261189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.261193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146271104:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699265.261201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271232:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699265.261294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.261309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace93800 x1874531146271232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.261321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146271232/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.261328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.261332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.261335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271232:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699265.261342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271360:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699265.261383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.261406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace91880 x1874531146271360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.261417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146271360/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.261424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.261428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.261431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271360:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699265.261437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271488:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699265.262670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.262698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace92680 x1874531146271488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.262714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146271488/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.262723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.262727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.262731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271488:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699265.262739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271616:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699265.262761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.262773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace91500 x1874531146271616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.262785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146271616/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.262794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.262797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.262800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146271616:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699265.262808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146271744:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699265.262830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.262841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace93480 x1874531146271744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.262852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146271744/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.262861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.262864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.262867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146271744:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699265.262872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146271872:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699265.262892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.262902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace92d80 x1874531146271872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.262913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146271872/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.262924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.262927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.262930:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146271872:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699265.262937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146272000:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699265.262959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.262969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace90e00 x1874531146272000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.262980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146272000/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.262987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.262990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.262993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146272000:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699265.263000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146272128:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699265.263020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699265.263030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699265/real 1787699265] req@ffff8f51ace93b80 x1874531146272128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699265.263042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146272128/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699265.263050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699265.263052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699265.263056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146272128:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699266.154566:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699267.178592:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699268.202543:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699269.227533:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699270.250734:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699270.378481:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378492:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699270.378497:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378507:0:75078:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378518:0:75078:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378524:0:75078:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699270.378561:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92300 x1874531146272256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378604:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378608:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699270.378610:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378614:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378618:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378621:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378630:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699270.378642:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93100 x1874531146272384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378651:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378655:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699270.378656:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378660:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378663:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378666:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378670:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699270.378680:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90700 x1874531146272512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378695:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378698:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699270.378700:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378703:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378706:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378709:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378714:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699270.378723:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90000 x1874531146272640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378732:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378735:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699270.378736:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378739:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378742:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378745:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378749:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699270.378765:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91180 x1874531146272768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378774:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378777:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699270.378779:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378783:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378786:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378790:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378795:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699270.378805:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91c00 x1874531146272896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378814:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378827:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699270.378828:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378832:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378835:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378838:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378843:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699270.378852:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146273024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378861:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378864:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699270.378865:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378868:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378871:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378874:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378877:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699270.378884:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146273152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699270.378899:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699270.378902:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699270.378904:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699270.378908:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7170 00000100:00080000:0.0:1787699270.378911:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699270.378914:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699270.378917:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699270.378926:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146273280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699270.379017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146272256:192.168.201.125@tcp:250:kworker.0 00000800:00000100:3.0:1787699270.379212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace92300 x1874531146272256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92300 x1874531146272256/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146272256:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699270.379310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272384:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699270.379331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace93100 x1874531146272384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93100 x1874531146272384/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272384:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699270.379397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272512:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699270.379417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace90700 x1874531146272512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90700 x1874531146272512/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272512:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699270.379472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272640:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699270.379489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace90000 x1874531146272640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90000 x1874531146272640/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272640:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699270.379527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272768:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699270.379543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace91180 x1874531146272768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91180 x1874531146272768/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146272768:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699270.379584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146272896:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699270.379599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace91c00 x1874531146272896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91c00 x1874531146272896/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146272896:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699270.379638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146273024:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699270.379653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace93b80 x1874531146273024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146273024/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146273024:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699270.379693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146273152:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699270.379710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace90e00 x1874531146273152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146273152/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146273152:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699270.379765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146273280:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699270.379781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699270.379791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699270/real 1787699270] req@ffff8f51ace92d80 x1874531146273280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699270.379801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146273280/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699270.379808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699270.379811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699270.379815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146273280:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699271.276560:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699272.298572:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699273.322794:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699274.346677:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699275.370659:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699275.498621:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498632:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699275.498636:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498646:0:75078:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498657:0:75078:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498664:0:75078:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699275.498699:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146273408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.498725:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498728:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699275.498730:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498735:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498739:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498743:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.498753:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699275.498765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146273408:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787699275.498769:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146273536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.498782:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498786:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699275.498788:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498792:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498796:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498800:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.498805:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699275.498817:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146273664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787699275.498820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699275.498832:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498836:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699275.498838:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498842:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498846:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498850:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.498855:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699275.498862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183647800 x1874531146273408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699275.498869:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146273792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.498884:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498887:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699275.498889:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498892:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498895:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498898:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.498902:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699275.498910:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146273920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.498918:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498921:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699275.498923:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498926:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498929:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498932:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.498936:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699275.498947:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146274048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.498961:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.498964:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699275.498966:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.498970:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.498972:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.498974:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.498986:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699275.498997:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146274176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.499006:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.499008:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699275.499010:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.499013:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.499017:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.499020:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.499024:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699275.499034:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146274304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699275.499048:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699275.499051:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699275.499052:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699275.499055:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7175 00000100:00080000:1.0:1787699275.499058:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699275.499061:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699275.499065:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699275.499074:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146274432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699275.499152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146273408/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146273408:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699275.499191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273536:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699275.499222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183646680 x1874531146273536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146273536/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273536:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699275.499260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273664:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699275.499273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183644000 x1874531146273664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146273664/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273664:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699275.499311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273792:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699275.499335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183645c00 x1874531146273792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146273792/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273792:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699275.499389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273920:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699275.499403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183645180 x1874531146273920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146273920/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146273920:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699275.499446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274048:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699275.499465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183647100 x1874531146274048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146274048/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274048:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699275.499505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274176:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699275.499523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183647480 x1874531146274176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146274176/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274176:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699275.499549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274304:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699275.499570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183644380 x1874531146274304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146274304/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274304:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699275.499609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274432:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699275.499623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699275.499632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699275/real 1787699275] req@ffff8f5183644e00 x1874531146274432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699275.499642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146274432/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699275.499660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699275.499663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699275.499666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146274432:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699276.394692:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699277.418992:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699278.442568:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699279.467456:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699280.491682:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699280.619092:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.619103:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699280.619107:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.619117:0:75078:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.619127:0:75078:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.619135:0:75078:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699280.619171:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646d80 x1874531146274560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699280.619257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146274560:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699280.621200:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.621208:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699280.621211:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.621218:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.621227:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.621232:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.621242:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699280.621270:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646a00 x1874531146274688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699280.627962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146274688:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699280.628147:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.628153:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699280.628156:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.628161:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.628167:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.628173:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.628183:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699280.628211:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146274816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699280.628226:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.628229:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699280.628231:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.628234:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.628237:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.628240:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.628245:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699280.628254:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146274944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699280.628270:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.628273:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699280.628275:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.628279:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.628281:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.628285:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.628288:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699280.629017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146274816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699280.629054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146274944:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699280.629133:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146275072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699280.629152:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.629155:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699280.629157:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.629161:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.629165:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.629169:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.629175:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699280.629186:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146275200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699280.629196:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.629200:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699280.629202:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.629205:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.629208:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.629211:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.629216:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699280.629224:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146275328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699280.629232:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.629235:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699280.629237:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.629240:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.629242:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.629245:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.629249:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699280.629258:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146275456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699280.629267:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699280.629271:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699280.629272:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699280.629276:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7180 00000100:00080000:1.0:1787699280.629279:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699280.629282:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699280.629286:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699280.629943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146275072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699280.629973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699280.629991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699280.630010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699280.630237:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645180 x1874531146275584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699280.630299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275584:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699281.515456:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699282.538564:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699283.562650:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787699283.693224:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.125:988 00000400:00000100:2.0:1787699283.693270:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.125@tcp at host 192.168.201.125 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787699283.693286:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787699283.693331:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000800:00000100:2.0:1787699283.693344:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787699283.693351:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:3.0:1787699283.697167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183645180 x1874531146275584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699283.697208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645180 x1874531146275584/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699283.697220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699283.697226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699283.697230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275584:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699283.697243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183646a00 x1874531146274688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699283.697256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646a00 x1874531146274688/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699283.697264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699283.697268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699283.697271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146274688:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699283.697283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183646d80 x1874531146274560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699283.697296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646d80 x1874531146274560/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699283.697303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699283.697307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699283.697310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146274560:192.168.201.125@tcp:250:kworker.0 00000800:00000100:2.0:1787699283.704038:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787699283.704059:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787699283.704073:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787699283.704085:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000800:00000100:2.0:1787699283.704095:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.125@tcp 00000100:00100000:1.0:1787699283.704188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183644e00 x1874531146275072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.704227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146275072/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.704241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.704248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699283.704252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146275072:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699283.704261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183644380 x1874531146275200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.704273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146275200/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.704282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.704286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699283.704290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275200:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699283.704298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183647480 x1874531146275328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.704310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146275328/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.704317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.704321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699283.704325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275328:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699283.704333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183647100 x1874531146275456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.704345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146275456/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.704351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.704380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699283.704384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146275456:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699283.704392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183647b80 x1874531146274816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.704405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146274816/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.704413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.704416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699283.704419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146274816:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699283.704425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699280/real 1787699283] req@ffff8f5183644700 x1874531146274944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699283.704433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146274944/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699283.704440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699283.704443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699283.704445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146274944:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699284.586613:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699285.610513:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699285.738467:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738477:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699285.738481:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699285.738543:0:75078:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738554:0:75078:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.738561:0:75078:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699285.738593:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644700 x1874531146275712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699285.738617:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738619:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699285.738621:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699285.738624:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738626:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.738629:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699285.738636:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699285.738644:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146275840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699285.738651:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738653:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699285.738655:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699285.738657:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738660:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699285.738664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146275712:192.168.201.125@tcp:250:kworker.0 00000100:00080000:1.0:1787699285.738665:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699285.738671:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699285.738678:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146275968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699285.738688:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738690:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699285.738692:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699285.738697:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738700:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.738704:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699285.738709:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699285.738721:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146276096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787699285.738726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699285.738734:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738750:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699285.738752:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699285.738757:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738762:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.738769:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699285.738769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183644700 x1874531146275712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699285.738778:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699285.738791:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146276224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699285.738825:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738828:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699285.738830:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787699285.738831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644700 x1874531146275712/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699285.738837:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738844:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699285.738848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699285.738851:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00080000:3.0:1787699285.738856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000008:00100000:1.0:1787699285.738860:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699285.738863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146275712:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699285.738873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146275840:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699285.738878:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146276352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787699285.738897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699285.738901:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738906:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699285.738909:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787699285.738913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183647b80 x1874531146275840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699285.738914:0:75078:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738921:0:75078:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.738928:0:75078:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699285.738934:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699285.738934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146275840/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.738947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787699285.738952:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146276480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699285.738954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.738961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146275840:192.168.201.125@tcp:38:kworker.0 00000100:00080000:1.0:1787699285.738971:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.738975:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699285.738977:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699285.738983:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.738987:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.738991:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699285.738993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146275968:192.168.201.125@tcp:38:kworker.0 00000008:00100000:1.0:1787699285.738997:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699285.739012:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146276608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787699285.739020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:1.0:1787699285.739029:0:75078:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699285.739033:0:75078:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699285.739037:0:75078:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787699285.739037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183647100 x1874531146275968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699285.739044:0:75078:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7185 00000100:00080000:1.0:1787699285.739050:0:75078:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699285.739056:0:75078:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146275968/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699285.739063:0:75078:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787699285.739074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699285.739081:0:75078:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146276736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699285.739084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146275968:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699285.739095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146276096:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699285.739114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183647480 x1874531146276096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699285.739134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146276096/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.739142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.739149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146276096:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699285.739154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146276224:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699285.739167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183644380 x1874531146276224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699285.739185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146276224/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.739193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.739199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146276224:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699285.739204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276352:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699285.739218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183644e00 x1874531146276352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699285.739235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146276352/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.739244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.739250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276352:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699285.739255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276480:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699285.739272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183645c00 x1874531146276480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699285.739293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146276480/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.739299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.739305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276480:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699285.739321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276608:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699285.739335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183644000 x1874531146276608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699285.739378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146276608/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.739387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.739392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276608:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699285.739397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276736:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699285.739413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699285.739422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699285/real 1787699285] req@ffff8f5183646680 x1874531146276736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699285.739432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146276736/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699285.739451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699285.739454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699285.739457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146276736:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699286.634574:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699287.660086:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699288.682661:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699289.706820:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699290.734085:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699290.858669:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.858679:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699290.858684:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.858694:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.858711:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.858751:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:2.0:1787699290.858806:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704e680 x1874531146276864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699290.858833:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.858837:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699290.858839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.858843:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.858847:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.858861:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.858872:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699290.858879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146276864:192.168.201.125@tcp:250:kworker.0 00000100:00100000:2.0:1787699290.858887:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704df80 x1874531146276992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699290.858899:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.858903:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699290.858906:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.858910:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.858913:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.858918:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.858923:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699290.858936:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704f800 x1874531146277120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699290.858956:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787699290.858959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:2.0:1787699290.858963:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699290.858966:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.858971:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.858976:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.858980:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.858986:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699290.858998:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704ce00 x1874531146277248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699290.859007:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.859010:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699290.859012:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787699290.859017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704e680 x1874531146276864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699290.859018:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.859024:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.859028:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.859033:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699290.859048:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c380 x1874531146277376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699290.859056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704e680 x1874531146276864/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699290.859062:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.859070:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699290.859072:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699290.859073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699290.859077:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.859082:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699290.859082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146276864:192.168.201.125@tcp:250:kworker.0 00000100:00080000:2.0:1787699290.859090:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.859098:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699290.859110:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c000 x1874531146277504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699290.859113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146276992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699290.859127:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.859130:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699290.859132:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.859136:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.859138:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.859142:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.859147:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787699290.859154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699290.859158:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704d180 x1874531146277632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699290.859168:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787699290.859171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704df80 x1874531146276992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699290.859172:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699290.859176:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.859184:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00080000:2.0:1787699290.859189:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699290.859193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704df80 x1874531146276992/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699290.859194:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.859202:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787699290.859209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787699290.859215:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704fb80 x1874531146277760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699290.859217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146276992:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699290.859233:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699290.859238:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699290.859239:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699290.859244:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7190 00000100:00100000:3.0:1787699290.859245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146277120:192.168.201.125@tcp:38:kworker.0 00000100:00080000:2.0:1787699290.859248:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787699290.859263:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:2.0:1787699290.859269:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787699290.859274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:2.0:1787699290.859282:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518704c700 x1874531146277888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699290.859291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704f800 x1874531146277120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704f800 x1874531146277120/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146277120:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699290.859326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146277248:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699290.859344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699290.859380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704ce00 x1874531146277248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704ce00 x1874531146277248/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146277248:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699290.859406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146277376:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699290.859420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699290.859426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704c380 x1874531146277376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c380 x1874531146277376/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146277376:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699290.859464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277504:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699290.859482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699290.859493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704c000 x1874531146277504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c000 x1874531146277504/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277504:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699290.859523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277632:192.168.201.125@tcp:38:kworker.0 00000800:00000100:3.0:1787699290.859546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699290.859556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704d180 x1874531146277632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704d180 x1874531146277632/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277632:192.168.201.125@tcp:38:kworker.0 00000100:00100000:3.0:1787699290.859583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277760:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699290.859594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699290.859600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704fb80 x1874531146277760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704fb80 x1874531146277760/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277760:192.168.201.125@tcp:8:kworker.0 00000100:00100000:3.0:1787699290.859617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277888:192.168.201.125@tcp:8:kworker.0 00000800:00000100:3.0:1787699290.859631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699290.859640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff8f518704c700 x1874531146277888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699290.859651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518704c700 x1874531146277888/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.859658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699290.859673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:3.0:1787699290.859676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518704c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146277888:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699291.757938:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699292.779673:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699293.802536:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699294.826784:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699295.851229:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699295.978896:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.978926:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699295.978932:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.978945:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.978958:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.978975:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699295.979026:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146278016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979070:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979074:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699295.979076:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979087:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979091:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979095:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979104:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699295.979117:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146278144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979134:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979137:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699295.979139:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979142:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979146:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979156:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979160:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699295.979170:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146278272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979184:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979188:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699295.979189:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979193:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979196:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979200:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979204:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699295.979222:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146278400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979229:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979233:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699295.979234:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979237:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979241:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979244:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979248:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699295.979265:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146278528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979274:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979278:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699295.979279:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979283:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979286:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979290:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979301:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699295.979312:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146278656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979321:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979332:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699295.979333:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979337:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979340:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979344:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979349:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699295.979399:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146278784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979414:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979417:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699295.979419:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979422:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979426:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979434:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979440:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699295.979457:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146278912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699295.979465:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699295.979468:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699295.979470:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699295.979474:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7195 00000100:00080000:3.0:1787699295.979482:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699295.979486:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699295.979491:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699295.979501:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146279040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699295.979615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146278016:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787699295.979722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.979785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f5187388380 x1874531146278016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.979814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146278016/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.979823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.979827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.979835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146278016:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787699295.979843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278144:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699295.979868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.979886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f518738bb80 x1874531146278144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.979898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146278144/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.979910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.979913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.979916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278144:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699295.979922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278272:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699295.979944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.979962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f518738b800 x1874531146278272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.979979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146278272/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.979991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.979994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.980001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278272:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699295.980006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278400:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699295.980028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.980038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f5187388a80 x1874531146278400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.980048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146278400/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.980059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.980062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.980065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278400:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699295.980070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278528:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699295.980090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.980099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f518738b480 x1874531146278528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.980114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146278528/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.980121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.980123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.980126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146278528:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699295.980131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146278656:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699295.980150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.980165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f5187389500 x1874531146278656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.980175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146278656/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.980188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.980191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.980194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146278656:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699295.980199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146278784:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699295.980221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.980250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f5187389c00 x1874531146278784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.980263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146278784/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.980270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.980273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.980276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146278784:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699295.980282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146278912:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699295.980301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.980322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f5187389180 x1874531146278912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.980333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146278912/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.980345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.980348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.980350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146278912:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699295.981997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146279040:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699295.982095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699295.982132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff8f518738aa00 x1874531146279040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.982157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146279040/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.982167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699295.982171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699295.982180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146279040:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699296.875078:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699297.898525:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699298.922652:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699299.946621:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699300.970703:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699301.098449:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098461:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699301.098465:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.098476:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098487:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.098494:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699301.098534:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146279168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699301.098586:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098591:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699301.098593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.098597:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098602:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.098607:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.098728:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699301.098743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146279168:192.168.201.125@tcp:250:kworker.0 00000100:00100000:3.0:1787699301.098747:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388000 x1874531146279296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699301.098761:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699301.098766:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.098770:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098774:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.098778:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.098782:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699301.098793:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a680 x1874531146279424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699301.098802:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098805:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699301.098806:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.098810:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098813:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.098817:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.098822:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787699301.098827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:3.0:1787699301.098837:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389f80 x1874531146279552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699301.098853:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098856:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699301.098858:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.098863:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098866:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699301.098869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f518738ad80 x1874531146279168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699301.098873:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.098880:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699301.098894:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146279168/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699301.098896:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738a300 x1874531146279680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699301.098916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699301.098918:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098924:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699301.098925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00080000:3.0:1787699301.098927:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699301.098933:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146279168:192.168.201.125@tcp:250:kworker.0 00000100:00080000:3.0:1787699301.098935:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098943:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.098947:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.098952:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699301.098963:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b100 x1874531146279808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699301.098966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279296:192.168.201.125@tcp:38:kworker.0 00000100:00080000:3.0:1787699301.098981:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.098985:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699301.098987:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.098992:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.098996:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699301.098998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00080000:3.0:1787699301.099004:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.099009:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699301.099015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f5187388000 x1874531146279296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699301.099022:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388e00 x1874531146279936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699301.099033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388000 x1874531146279296/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699301.099035:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.099039:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699301.099040:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.099051:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.099055:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.099058:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.099062:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699301.099071:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738ad80 x1874531146280064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699301.099078:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699301.099081:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699301.099083:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699301.099086:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7200 00000100:00080000:3.0:1787699301.099089:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699301.099091:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699301.099095:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699301.099104:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146280192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699301.100623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.100629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.100634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279296:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699301.100642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279424:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699301.100686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.100705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f518738a680 x1874531146279424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.100719:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a680 x1874531146279424/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.100732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.100735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.100738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279424:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699301.100745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279552:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699301.100767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.100793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f5187389f80 x1874531146279552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.100805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389f80 x1874531146279552/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.100813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.100816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.100819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279552:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699301.100825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279680:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699301.100845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.100856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f518738a300 x1874531146279680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.100868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738a300 x1874531146279680/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.100875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.100878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.100881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146279680:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699301.100901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146279808:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699301.100920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.100930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f518738b100 x1874531146279808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.100941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b100 x1874531146279808/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.100950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.100953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.100956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146279808:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699301.100961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146279936:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699301.100979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.100990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f5187388e00 x1874531146279936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.101000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388e00 x1874531146279936/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.101007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.101010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.101013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146279936:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699301.101018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146280064:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699301.101031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.101041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f518738ad80 x1874531146280064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.101052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738ad80 x1874531146280064/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.101059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.101062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.101065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146280064:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699301.101071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146280192:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699301.101089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699301.101099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699301/real 1787699301] req@ffff8f518738aa00 x1874531146280192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699301.101110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146280192/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699301.101117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699301.101120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699301.101123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146280192:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699302.046134:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699303.386707:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699304.299026:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699305.322664:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699306.218633:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218644:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699306.218648:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218659:0:14956:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218673:0:14956:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218680:0:14956:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:1.0:1787699306.218717:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647800 x1874531146280320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218749:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218754:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699306.218756:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218760:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218764:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218768:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.218778:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699306.218793:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183646680 x1874531146280448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218802:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218805:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699306.218808:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218811:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218815:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218818:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.218822:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699306.218833:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644000 x1874531146280576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218847:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218850:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699306.218851:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218855:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218857:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218861:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.218866:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699306.218876:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183645c00 x1874531146280704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218884:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218888:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699306.218889:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218892:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218895:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218897:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.218901:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699306.218911:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644e00 x1874531146280832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218919:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218922:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699306.218923:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218927:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218930:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218934:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.218937:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699306.218947:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183644380 x1874531146280960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218955:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218958:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699306.218959:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218962:0:14956:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.218966:0:14956:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.218968:0:14956:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.218972:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787699306.218981:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647480 x1874531146281088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.218989:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.218992:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699306.218994:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.218997:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.219000:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.219003:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.219006:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699306.219015:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647100 x1874531146281216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699306.219028:0:14956:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699306.219031:0:14956:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699306.219032:0:14956:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699306.219035:0:14956:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7205 00000100:00080000:1.0:1787699306.219039:0:14956:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787699306.219042:0:14956:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:1.0:1787699306.219046:0:14956:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787699306.219055:0:14956:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5183647b80 x1874531146281344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699306.219128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146280320:192.168.201.125@tcp:250:kworker.0 00000800:00000100:0.0:1787699306.219187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183647800 x1874531146280320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647800 x1874531146280320/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146280320:192.168.201.125@tcp:250:kworker.0 00000100:00100000:0.0:1787699306.219279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280448:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699306.219302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183646680 x1874531146280448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183646680 x1874531146280448/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183646680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280448:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699306.219344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280576:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699306.219385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183644000 x1874531146280576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644000 x1874531146280576/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280576:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699306.219436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280704:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699306.219455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183645c00 x1874531146280704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183645c00 x1874531146280704/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183645c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280704:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699306.219511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280832:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699306.219531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183644e00 x1874531146280832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644e00 x1874531146280832/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146280832:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699306.219572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146280960:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699306.219591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183644380 x1874531146280960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183644380 x1874531146280960/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183644380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146280960:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699306.219630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146281088:192.168.201.125@tcp:38:kworker.0 00000800:00000100:0.0:1787699306.219649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183647480 x1874531146281088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647480 x1874531146281088/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146281088:192.168.201.125@tcp:38:kworker.0 00000100:00100000:0.0:1787699306.219689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146281216:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699306.219708:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183647100 x1874531146281216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647100 x1874531146281216/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146281216:192.168.201.125@tcp:8:kworker.0 00000100:00100000:0.0:1787699306.219750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146281344:192.168.201.125@tcp:8:kworker.0 00000800:00000100:0.0:1787699306.219768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:0.0:1787699306.219778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699306/real 1787699306] req@ffff8f5183647b80 x1874531146281344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699306.219788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5183647b80 x1874531146281344/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699306.219795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699306.219797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:0.0:1787699306.219800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5183647b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146281344:192.168.201.125@tcp:8:kworker.0 00010000:00010000:0.0:1787699306.346728:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699307.374682:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699308.394642:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699309.418568:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699310.442678:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699311.338576:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338588:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699311.338593:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338606:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338620:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338629:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:0.0:1787699311.338686:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93480 x1874531146281472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.338723:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338727:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699311.338729:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338733:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338736:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338740:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.338750:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699311.338761:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91500 x1874531146281600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.338771:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338774:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699311.338776:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338779:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338781:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338784:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.338787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699311.338795:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92680 x1874531146281728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.338803:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338807:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699311.338809:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338812:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338815:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338818:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.338821:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699311.338828:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace91880 x1874531146281856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.338835:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338838:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699311.338839:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338842:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338845:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338848:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.338852:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699311.338860:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93800 x1874531146281984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.338868:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338871:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699311.338872:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338876:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338878:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338881:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.338886:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699311.338894:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90380 x1874531146282112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.338901:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.338965:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699311.338967:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.338970:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.338974:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.338977:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.338981:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699311.338994:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace92d80 x1874531146282240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.339002:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.339006:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699311.339008:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.339011:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.339014:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.339017:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.339020:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699311.339027:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace90e00 x1874531146282368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699311.339035:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699311.339037:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699311.339039:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699311.339043:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7210 00000100:00080000:0.0:1787699311.339046:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787699311.339049:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:0.0:1787699311.339053:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699311.339060:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f51ace93b80 x1874531146282496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699311.339166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146281472:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787699311.339241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace93480 x1874531146281472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93480 x1874531146281472/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146281472:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787699311.339335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281600:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699311.339382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace91500 x1874531146281600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91500 x1874531146281600/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281600:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699311.339428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281728:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699311.339445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace92680 x1874531146281728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92680 x1874531146281728/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281728:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699311.339481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281856:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699311.339496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace91880 x1874531146281856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace91880 x1874531146281856/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281856:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699311.339532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281984:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699311.339546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace93800 x1874531146281984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93800 x1874531146281984/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146281984:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699311.339582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282112:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699311.339595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace90380 x1874531146282112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90380 x1874531146282112/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282112:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699311.339631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282240:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699311.339644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace92d80 x1874531146282240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace92d80 x1874531146282240/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace92d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282240:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699311.339677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282368:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699311.339690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace90e00 x1874531146282368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace90e00 x1874531146282368/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282368:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699311.339725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282496:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699311.339738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699311.339747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699311/real 1787699311] req@ffff8f51ace93b80 x1874531146282496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699311.339757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f51ace93b80 x1874531146282496/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699311.339764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699311.339767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699311.339770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f51ace93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146282496:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699311.468073:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699312.490675:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699313.514823:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699314.541020:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787699315.566739:0:11606:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699316.458608:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 2b50556e-3530-46e3-af38-7a9a83888129->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458620:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699316.458626:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458640:0:57472:0:(import.c:510:import_select_connection()) MGC192.168.201.125@tcp: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458653:0:57472:0:(import.c:589:import_select_connection()) MGC192.168.201.125@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458662:0:57472:0:(import.c:624:import_select_connection()) MGC192.168.201.125@tcp: import ffff8f5190afe800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000100:00100000:3.0:1787699316.458724:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389180 x1874531146282624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458760:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458764:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699316.458766:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458771:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458774:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458778:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518356d800: import ffff8f518a176000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.458787:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699316.458800:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389c00 x1874531146282752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458809:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458812:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699316.458814:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458817:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458820:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458824:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518356d800: import ffff8f5191d0e800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.458829:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699316.458839:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187389500 x1874531146282880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458848:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458851:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699316.458852:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458855:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458858:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458861:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518356d800: import ffff8f5191d0b000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.458865:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699316.458877:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b480 x1874531146283008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458886:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) 46e6b40d-f674-4e7b-a698-924b4152ad65->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458889:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699316.458890:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458894:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458897:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458901:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518356d800: import ffff8f5191d08800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.458905:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699316.458915:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388a80 x1874531146283136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458923:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458926:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699316.458928:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458932:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458935:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458939:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8f518719a000: import ffff8f5184593000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.458943:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699316.458953:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738b800 x1874531146283264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458961:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458964:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699316.458965:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.458969:0:57472:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.458971:0:57472:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.458974:0:57472:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8f518719a000: import ffff8f5190afc000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.458978:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787699316.458987:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738bb80 x1874531146283392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.458995:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.458998:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699316.459000:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.459003:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.459006:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.459009:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8f518719a000: import ffff8f5190afb000 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.459013:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699316.459023:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f5187388380 x1874531146283520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699316.459031:0:57472:0:(pinger.c:229:ptlrpc_pinger_process_import()) e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699316.459034:0:57472:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699316.459035:0:57472:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699316.459039:0:57472:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: connect to NID 192.168.201.125@tcp last attempt 7216 00000100:00080000:3.0:1787699316.459041:0:57472:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699316.459045:0:57472:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8f518719a000: import ffff8f5187449800 using connection 192.168.201.125@tcp/192.168.201.125@tcp 00000008:00100000:3.0:1787699316.459048:0:57472:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699316.459069:0:57472:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8f518738aa00 x1874531146283648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787699316.459191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146282624:192.168.201.125@tcp:250:kworker.0 00000800:00000100:1.0:1787699316.459255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.459306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f5187389180 x1874531146282624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.459320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389180 x1874531146282624/t0(0) o250->MGC192.168.201.125@tcp@192.168.201.125@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.459327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afe800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.459329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.459332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:2b50556e-3530-46e3-af38-7a9a83888129:2398:1874531146282624:192.168.201.125@tcp:250:kworker.0 00000100:00100000:1.0:1787699316.459336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146282752:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699316.459396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.459410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f5187389c00 x1874531146282752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.459429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389c00 x1874531146282752/t0(0) o38->lustre-MDT0000-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.459437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f518a176000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.459440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.459444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146282752:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699316.459449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146282880:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699316.459903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.459917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f5187389500 x1874531146282880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.459930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187389500 x1874531146282880/t0(0) o38->lustre-MDT0001-mdc-ffff8f518356d800@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.459938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0e800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.459942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.459945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187389500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146282880:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699316.459951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146283008:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699316.459971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.459981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f518738b480 x1874531146283008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.459992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b480 x1874531146283008/t0(0) o8->lustre-OST0000-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.460006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d0b000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.460010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.460013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146283008:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699316.460019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146283136:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699316.460038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.460048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f5187388a80 x1874531146283136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.460059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388a80 x1874531146283136/t0(0) o8->lustre-OST0001-osc-ffff8f518356d800@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.460066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5191d08800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.460069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.460072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:46e6b40d-f674-4e7b-a698-924b4152ad65:2398:1874531146283136:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699316.460078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283264:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699316.460096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.460106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f518738b800 x1874531146283264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.460117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738b800 x1874531146283264/t0(0) o38->lustre-MDT0000-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.460125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5184593000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.460129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.460132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283264:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699316.460137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283392:192.168.201.125@tcp:38:kworker.0 00000800:00000100:1.0:1787699316.460155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.460165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f518738bb80 x1874531146283392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.460176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738bb80 x1874531146283392/t0(0) o38->lustre-MDT0001-mdc-ffff8f518719a000@192.168.201.125@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.460181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afc000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.460183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.460187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283392:192.168.201.125@tcp:38:kworker.0 00000100:00100000:1.0:1787699316.460192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283520:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699316.460210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.460217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f5187388380 x1874531146283520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.460224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f5187388380 x1874531146283520/t0(0) o8->lustre-OST0000-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.460228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5190afb000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.460230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.460232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f5187388380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283520:192.168.201.125@tcp:8:kworker.0 00000100:00100000:1.0:1787699316.460236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283648:192.168.201.125@tcp:8:kworker.0 00000800:00000100:1.0:1787699316.460246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.125@tcp 00000100:00100000:1.0:1787699316.460251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699316/real 1787699316] req@ffff8f518738aa00 x1874531146283648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699316.460258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8f518738aa00 x1874531146283648/t0(0) o8->lustre-OST0001-osc-ffff8f518719a000@192.168.201.125@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699316.460262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff8f5187449800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699316.460263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.125@tcp failed (-110) 00000100:00100000:1.0:1787699316.460265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8f518738aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e4c6a2c0-e8d9-4fef-9bc8-45d8e526a9bd:2398:1874531146283648:192.168.201.125@tcp:8:kworker.0 00010000:00010000:1.0:1787699316.586841:0:6169:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699317.611879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787699318.634622:0:34679:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 75140 lines, 75140 kept, 0 dropped, 0 bad.